Student Explanations and Patterns of Use In a Hypermedia Learning Environment
Tricia Jones, University of Michigan
Gail P. Baxter, Educational Testing Service

Abstract
"Seeing Through Chemistry" is a multi-faceted hypermedia environment designed to support student learning in an introductory college chemistry course. Design features such as content coverage, inquiry questions, and feedback provide students opportunities to make connections among concepts as they develop an understanding of chemical phenomena. In this paper, we consider the impact of these design features on student explanations (proxy for student learning) for the inquiry questions in two of the six instructional modules and their influence on students’ patterns of use. Results indicate that feedback on answer correctness and provision of "model" explanations led students to revise their explanations. Nevertheless the overall quality of explanations was generally low for all inquiry questions; they did not seem to depend on the content coverage. Implications for hypermedia learning environment design are discussed.

1. Introduction
Learning is a process of constructing new knowledge on the basis of current knowledge. As learning occurs, increasingly well-structured and qualitatively different organizations of knowledge develop. Structured knowledge is not just a consequence of the amount of information received, but reflects exposure to an environment for learning where there are opportunities for problem solving, interpreting, working in unfamiliar environments requiring transfer, and making connections between relevant concepts. The inference to be made for the design of effective learning environments is that how you learn is as important as what you learn.

Hypermedia systems may be particularly appropriate as learning environments because the associative knowledge structures of a topic or subject matter are made explicit. However, in creating hypermedia systems, attention must also be given to design features that can support students in making appropriate connections in their own developing knowledge structures (e.g., McKendree et al 1995). Activities that will emphasize the key concepts of a domain and engage the students in problem solving and reflection are especially important in this regard. Examples include guided tours highlighting concepts in a domain and showing how they are related; multiple paths through materials that encourage consideration of the same concepts in differing contexts; questions that promote reflection on the relationships among concepts; and opportunities to generate and revise explanations of key concepts as understanding develops. The inclusion of these and other design features in hypermedia learning environments may have implications for the ways in which the software is used and for the quality of learning that results.

In this paper, we consider the impact of various design features on student learning and patterns of use in a multi-faceted hypermedia environment, "Seeing Through Chemistry" [STC]. The theoretical and empirical basis for designing the STC learning environment (Dershimer et al. 1992) stems from psychological principles of learning such as cognitive flexibility theory (Spiro et al. 1988), supplemented by research on difficulties students have learning chemistry (Lythcott 1990; McDermott 1988), and supported by human-computer interaction guidelines. Cognitive Flexibility Theory posits that individuals learn most effectively when they "experience" concepts and ideas from a variety of perspectives. By revisiting content from different perspectives, the individual develops a rich understanding of a concept and its relationship to other concepts. This deep understanding supports thinking, reasoning, and inference making or what might be termed the flexible use of knowledge (Spiro et al. 1988).

Cognitive flexibility theory does not presuppose a computer-based presentation, but hypermedia and multimedia environments are well suited to the provision of learning opportunities that focus on knowledge construction and use. For example, STC integrates video, animations, and experimental simulations with text and pictures onto multimedia content cards to facilitate the development of students’ understanding of topics in an introductory college chemistry course (Dershimer & Rasmussen 1990). One goal of the STC software is to enrich students’ experiential background and allow them to "see" scientific phenomena. Some students lack laboratory experiences and may have never seen a violent acid reaction or light bending through a prism. Without a common experience base, they may not understand topics discussed in introductory college chemistry lectures. Consequently, STC provides students a visual representation of chemical phenomenon through a rich variety of multimedia components (still pictures, animations, video, simulations, and focus questions). Four interaction modes (self-guided inquiry, directed inquiry, structured inquiry, and guided inquiry), distinguished by the level of structure and support, focus students’ attention on the relationship among topics in each of the six modules and allow for multiple paths through the material to support individual interest and use.

An interactive demo (80 KB) of a part of the Acids & Bases module. (Shockwave plugin required)

STC’s designers created inquiry questions to provide scaffolding for students who need help building a mental structure or representation of the material (the directed inquiry mode). Students who did not need that help could browse the material through the concept map and standard navigational features (self-directed inquiry). As such, the inquiry questions were not intended as an assessment of student understanding. Rather, they provide an activity to help the student construct an understanding of foundational chemistry concepts and phenomena. In practice, however, inquiry questions provide the primary instructional activity in STC. They can be quite useful in this regard, because the questions represent a synthesis of topics or content cards, and the same topics may be relevant for several questions. Revisiting the same topics (e.g., concentration) in multiple contexts (e.g., pH, precipitation) should promote students’ understanding of the relationship among related concepts. Previous research has shown that "criss-crossing" concepts and topics facilitates connections among ideas (i.e., knowledge construction) (Jacobson 1991; Spiro et al. 1991) and the quality of students’ explanations of these concepts is directly related to their level of knowledge and understanding (e.g., Chi et al. 1989; Chi & van Lehn 1991).

Throughout STC, students make decisions on how the software is to be used – whether and when to read the content cards or view supplemental media; which of the four navigation modes to use; and whether and when to revise their explanations. How students respond to features of the software is referred to as patterns of use. Patterns of use are particularly relevant in hypermedia learning environments, where the student has the flexibility to make choices in interacting with the software, because these choices may impact the extent of learning. We cannot assume that all students have seen the same material.

In what follows, we consider two STC modules and examine the quality of student explanations provided in response to the inquiry questions. We chose these modules because they vary with respect to the opportunities for criss-crossing content cards when responding to inquiry questions. We expect that students will learn more (provide higher quality explanations) from instructional modules that provide more opportunities for criss-crossing. Our analysis of student answers and explanations to the inquiry questions provides a general measure of the impact of this design feature on student learning. Next, we examined student patterns of use and their relationship to students’ explanations to inquiry questions (used as a proxy for student learning or understanding) to elaborate on these general findings. We conclude with recommendations for software designers with regard both to interaction models and content design.

An interactive demo (21 KB) on an inquiry question.
(Shockwave plugin required)

Screenshots showing a concept map and the "criss-crossing" concepts for: the Acids & Bases module (20 KB) and the Solubility module (24 KB). (Click on the screenshot to toggle the concept map between with and without the illustration of "criss-crossing".)

2. Methods
2.1. Participants and Context
This study is part of a larger research project examining the use of Seeing Through Chemistry in an introductory chemistry course at a large Midwestern research university. Use of STC was required, but participation in the study was not: 201 of the 450 students enrolled in this multi-section lecture course volunteered to participate. Of the 201 participants, 180 completed the modules of interest in this study.

The semester these data were collected, all students used the software regularly, for graded assignments. Students answered the inquiry questions for five of the six modules in STC and turned in the printed version of their responses. The printout shows the answer choice and the explanation (initial and final). Teaching assistants checked the assignments for completeness. Students received up to 6 points on each assignment for a maximum possible of 30 points (5% of the overall course grade).

2.2. Study Design

Each of the six modules in STC, although designed with the same fundamental instructional goal, varies in terms of the content, the amount of coverage given to a specific topic within a module, and the opportunities for students to examine relationships among topics in a module. These module characteristics may influence how students interact with the material and the level of learning achieved. In order to examine that question, we set out to identify two modules that varied widely in coverage. The modules selected for the study are shown in Figure 1. Acids & Bases has 6 inquiry questions, and was the fourth module completed in the semester. Solubility has 10 questions, and was the fifth and final module completed.

For each of the STC modules, we constructed visualizations that identify the set of topics suggested for each inquiry question. The goal of creating these visualizations was to get a picture of the opportunities the software provides for "criss-crossing" or making connections among topics (i.e., topic coverage). We expect that more opportunities for examining relationships among topics (i.e., criss-crossing) will result in higher levels of understanding (i.e., better quality explanations). Ideally,

(a) all topics would be visited by a student prior to completing a particular module (minimally complete topic coverage),

(b) each topic would be visited for more than one inquiry question (repeat topic coverage), and

(c) a number of the same topics would be suggested for various inquiry questions (overlap of topic coverage).

Reading across the rows of Table 1, bullets (•) indicate when a topic (one or more content cards) is suggested for each inquiry question in a module. For example, in the Acids & Bases module, STC suggests topics 2-4, 4-1, 4-2, and 4-3 are relevant for answering inquiry question 1(topics are named in the software, but names have been omitted for the purposes of this discussion).

Table 1. Coverage in Acids & Bases and Solubility modules.

2.3 Measures
2.3.1 Log Files and Inquiry Question Responses
STC automatically captures the sequence and duration of choices made by the student. This information is recorded in files known as log files, one per student. Each log file captures completely a student’s use of STC throughout the semester including navigation choices, content selection, answers to inquiry questions, and revisions to explanations. This information can be used to identify patterns of use (what students do), characterize the quality of student explanations to inquiry questions (how well they perform), and thus examine the relationship between patterns of use and performance.

2.3.2 Patterns of Use
A variety of software tools derive patterns of use from the participants’ log files. Moves are extracted from the log files, and characterized by type (e.g., view question (q), view content card (c), edit response (r)). The sequence of moves is combined and then abstracted to form a "pattern of use" and a corresponding label is assigned to each inquiry question (Jones & Jones 1997).

Previous research (Jones, Berger, & Magnusson 1996) suggests the importance of distinguishing between base patterns and revision patterns. Base patterns represent the students’ sequence prior to entering their initial response, and indicate when they look at content cards relative to the question and the response. In using STC, students exhibit one of four base patterns. These are:

Questions only (qr): read inquiry question, respond

Cards first (cqr): read content cards, read inquiry question, respond

Question guided (qcr): read inquiry question, read content cards, respond

Cards before & within (cqcr): read content cards, read inquiry question, read content cards, respond

Revision patterns characterize students’ performance after their initial response. Do students revise their explanation? Do students refer to content cards between entering their initial explanation and final revision? Base and revision patterns of use were identified for each of the 16 inquiry questions (6 for Acids and Bases and 10 for the Solubility module) for each of the 180 students.

2.3.3 Explanation Quality
STC provides feedback on the correctness of student responses to inquiry questions, but it does not evaluate the quality of students’ explanations. A post hoc scoring scheme was designed to focus on the completeness, coherence, and accuracy of student explanations because research in cognitive psychology suggests that the quality of one’s explanations are indicative of the level of one’s understanding (e.g., Chi et al. 1989; Chi, Feltovich, & Glaser 1981). Initial and final (revised) explanations were read and assigned scores from 0 to 4 as follows: 0–Inadequate; 1–Fragmented; 2–Partial, 3–Good, and 4–Elaborated (Table 1). For example, a student who mentions two isolated facts but doesn’t relate them to each other or to the question might be given a level 1 score (Fragmented). A student who mentions all the relevant concepts, including an explanation of "mechanism," and who adds ideas not found in the model response would receive a level 4 (Elaborated). The first author scored both initial and final explanations to the inquiry questions. A second rater scored a 25% random sample. Inter-rater reliability was 0.87 and 0.79 for Acids & Bases and Solubility, respectively.

3. Results

The design of instructional software determines which teaching and learning activities can be supported by the learning environment, and those activities in turn greatly influence the learning that occurs, through the nature and quality of the interaction they support. In research concerning instructional learning environments, it is important to examine both whether the activities occur and the expected outcomes result (Ehrmann 1998).

We expect that better modules will lead students to better use of resources, and that a better use of resources will lead to better explanations. "Better modules" reflects instructional design. According to the framework of Cognitive Flexibility Theory, we define a better module as a more connected one, one that allows students more opportunities to revisit content. "Better use of resources" reflects student behavior. It implies reading and revisiting the content, revising explanations, taking advantage of support features, and so on. Intuitively, we expect those who read the content cards to write fuller and more precise explanations. On the other hand, perhaps some students do not read the cards prior to responding if they think they already know the material. They may intend for the exercise to serve as a "self check" on their understanding; only if they determine they don’t understand will they read further. Students may read the cards before they answer the question, or after they read a question but before responding. The former is like reading the chapter before doing the exercises; the latter is like pre-reading questions on a reading comprehension test. Both are legitimate strategies, but one may be more effective in this setting.

Because the design of the Solubility module is more robust and interconnected with respect to topic coverage than is Acids & Bases, we expected students to provide higher quality explanations for that module.

In analyzing student responses, we first considered the answer choices and the explanation quality. We further examined these measures in light of the process students undergo while answering inquiry questions, both what they do before making their initial response (base patterns of use) and what they do while working towards their final revision (revision patterns of use). We end with a discussion describing what additional features might influence revision patterns.

3.1 Answer Choices and Explanation Quality
For each inquiry question students were required to select an answer and generate an explanation. Answers were scored right or wrong. In general, the answer choices to inquiry questions were relatively easy for students (see Figure 2). In Acids & Bases, 80% or more of the students made the correct choice on 4 of the 6 inquiry questions. The most difficult question in this module was Question 4, with only 20% of the students selecting the correct choice. Similarly, the answer choices in the Solubility module were relatively easy. 55% or more of the students answered all of the Solubility questions correctly. 80% or more of the students answered 7 of the 10 solubility questions correctly. Furthermore, the majority of students answered more than half the questions correctly in each of the modules (i.e., 81% and 96% for Acids & Bases, and Solubility, respectively).

Figure 1. Answer correctness by question (n=180).

Explanations were scored on a 5 point scale from 0 to 4. In the Acids & Bases module, initial score totals range from 0 to 17 with a mean of 7.34 (sd=3.97). Totals after revisions range from 4 to 21, with a mean of 12.84 (sd=3.68). The means indicate that students average about 1 point per question (fragmented) for their initial explanations and about 2 points for their final explanation (partial). The change is statistically significant (t(df=179) = 18.65***). In the Solubility module, initial score totals range from 0 to 34, with a mean of 12.01 (sd=6.13). Final totals range from 6 to 36, with a mean of 19.06 (sd-6.44). With 10 questions in the module, this again means students improve from an average score just above 1 to just below 2. Again, the change is significant (t (df=179) = 15.40***).

In both modules we see that students generally select the correct answer, but they do not provide high quality explanations for their choices. Moreover, students do revise their explanations and the change in the quality of explanations is statistically significant but the magnitude of the difference is small and the result is still generally poor. Furthermore, contrary to expectations, there were no obvious differences between the two modules in overall quality. Therefore, we investigated if differences were related to student patterns of use. We characterized the various patterns of use, the consistency with which students use them within and between modules, and the relationship to explanation quality.

3.2 Base Patterns of Use
We first examined the base pattern components: Do students read cards first or questions first? Do students read cards in the process of answering questions before their initial response? In the Acids & Bases module, 73 students (40.6%) read some cards before reading any questions (shown by the top two patterns in Figure 3). In Solubility, 69 students (38.3%) did so. 61 students (34%) always read the cards first for both modules, and 20 students (11.1%) used different approaches for the two modules (read cards first for one module but not the other). The percentage of students who refer to cards while answering the questions ranges from 44% to 76% (the topmost and third patterns in Figure 3). The latter value indicates that a sizable number of students refer to cards for at least one question.

Figure 2. Base Patterns distribution, by question.

Glossary: To see the explanation of the abbreviations.

In both modules, the qcr pattern is used by almost half the students on each question. Although students are allowed to answer the questions in any order, many of them proceed in numeric order starting with question 1. Notice then that, as the module progresses, students tend to refer to cards between the question and the response (e.g. the size of the qcr group increases while qr decreases, and likewise for cqcr and cqr). This is contrary to expectations: we expected students would visit cards for earlier questions, and then not refer to them again.

In order to examine effectiveness of strategies, we examined mean scores within the base pattern groups. Initial explanation quality score means and standard deviations are shown in Table 2. (Only significant results are shown.) On each of the 16 questions, we conducted a within-subjects (base pattern group: qr vs. cqr vs. qcr vs. cqcr) ANOVA. Base pattern group effects were significant for 5 of the 16 questions. For 4 of these 5, significant group mean differences were found. For two questions (AB 1, Sol 7), the qr group had lower mean scores than all other groups. For a third question (Sol 10), the qr group had lower scores than the two cards-first groups (cqcr and cqr). Consistent with our expectations, these results suggest that patterns of use are related to performance. Students who read cards (either before the question or between the question and response) generate better quality explanations than students who don’t read cards prior to writing their response.

	Base Pattern of Use
Inquiry	qr		cqr		qcr		cqcr		sample
Question	M	(sd)	M	(sd)	M	(sd)	M	(sd)	M	(sd)	F	group differences
AB 1	0.98	(1.11)	1.87	(1.19)	1.81	(1.03)	2.24	(1.37)	1.65	(1.21)	8.47 *	qr < cqr, qcr, cqcr
Sol 2	1.02	(1.22)	1.66	(1.26)	0.62	(1.02)	1.73	(1.32)	1.12	(1.25)	9.58 *	qcr < cqr, cqcr
Sol 5	0.83	(1.29)	1.59	(1.30)	0.96	(1.06)	1.66	(1.14)	1.18	(1.22)	4.99 *
Sol 7	0.87	(0.97)	2.06	(0.89)	1.70	(0.90)	1.85	(0.94)	1.69	(0.97)	8.43 *	qr < cqr, qcr, cqcr
Sol 10	0.43	(0.99)	1.24	(1.16)	0.90	(0.86)	1.17	(0.95)	0.96	(0.98)	3.95 *	qr < cqcr, cqr

Table 3. Initial explanation quality score by base pattern group.

glossary: To see the explanation of the abbreviations.

Students revise, but at a different level depending on the question (see Figure 4). The percentages range from 20% to 90%, with about 40% to 60% on most questions. It is reasonable to expect students with incorrect choices (the solid blue bars) to revise, since if their choice was wrong it is highly likely their explanation is also wrong. However, even students with correct choices sometimes revise, as can be seen by comparing the size of the first (incorrect answers) and second (revision rates) bars for each question in Figure 4.

In both modules, there is one question where improvement is much less common. For Solubility question 3, only 42% of the revisers received a higher score. In contrast, 72% or more of the revisers improved their scores for all other questions in that module. Interestingly, Solubility question 3 did not have a model response – this perhaps suggests that many students rely on the model response in their revisions. In Acids & Bases, 43% of the revisers on question 5 do not see an improvement. The reason is not immediately obvious, but it might be that students were unable to pick out the crucial concepts in the model response and instead latched onto surface differences.

We wondered if students compare their explanation to the model response. As it turns out, one of the inquiry questions did not have a model response. An examination of responses to Solubility question 3 shows that the question was slightly more difficult than average; 80% of the students answered correctly. While all students with incorrect answers revised, only 20% of the students with correct answers revised. This is the lowest percentage of all the questions. Again, this suggests a reliance on the model response when revising answers.

According to the STC designers, revision should be a key process when students answer the inquiry questions (Dershimer et al. 1992). Substantial revision in the sense of improved explanations is not taking place. Why not? We see that students revise when the inquiry question choice is hard (when they choose the wrong answer). Perhaps the questions need to be harder or trickier. But also, the nature of the interaction – merely expecting students to revise – does not seem to encourage engagement in the content. Students need feedback on the quality of their explanations. A different response model can be seen in the Evolutionary Thematic Explorer (Jacobson 1997). In that system, students choose statements from a list and then have their answers critiqued with respect to two major theories of evolution. Experimental results show learning occurred. Perhaps similar activities here would help: students could assemble an explanation from sets of statements that either capture key concepts or highlight common misconceptions.

5. References 
Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., & Glaser, R. (1989). Self-explanation: How students study and use examples in learning to solve problems. Cognitive Science, 13(1):145–182.

Chi, M.T.H., Feltovich, P.J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science. 5(2):121–152.

Chi, M.T.H., & van Lehn, K.A. (1991). The content of physics self-explanations. The Journal of the Learning Sciences, 1(1):69–105.

Dershimer, R. C., & Rasmussen, P. G. (1990). Seeing Through Chemistry [Macintosh Multimedia Software]. Ann Arbor: Office of Instructional Technology, University of Michigan.

Dershimer, R. C., Wurman, P., Nowak, M. A., Saunders, E., Berger, C. F., & Rasmussen, P. G. (1992). Designing instructional hypermedia to facilitate concept development (unpublished manuscript, available from author).

Dershimer, R. C. (1996). Private communication.

Ehrmann, S. C. (1998). Studying teaching, learning and technology: A tool kit from the Flashlight Program. In Stephen C. Ehrmann and Robin Etter Zuñiga, The Flashlight Evaluation handbook (1.0). Washington, DC: The TLT Group. Also available online at http://www.tltgroup.org/resources/fstudtool.html (6 August 1999).

Jacobson, M.J. (1991). Knowledge acquisition, cognitive flexibility, and the instructional applications of hypertext: A comparison of contrasting designs for computer-enhanced learning environments. Unpublished doctoral dissertation, University of Illinois.

Jacobson, M.J. & Archodidou, A. (1997). Case- and problem-based hypermedia learning environments and conceptual change: The evolution of understandings about evolution. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.

Jones, T., Berger, C. F., & Magnusson, S. (1996). The pursuit of knowledge: Interviews and log files in hypermedia research. Educational Multimedia and Hypermedia Annual, 342–347. Charlottesville: AACE.

Jones, T., & Jones, M. C. (1997). MacSQEAL: A Tool for Exploration of Hypermedia Log File Sequences. Educational Multimedia and Hypermedia Annual, 539–544. Charlottesville: AACE.

Lythcott. (1990). Concept learning vs. problem solving: Revisited. Journal of Chemical Education, 64(6): 510–512.

McDermott, D. P. (1988). Chemistry and the framework of learning. Journal of Chemical Education, 65(6):539–540.

McKendree, J., Reader, W., & Hammond, N. (1995). The "Homeopathic Fallacy" in learning from hypertext. Interactions, 2(3):74–82.

Spiro, R. A., Coulson, R.L., & Feltovich, P.J. (1988). Cognitive flexibility theory: Advanced knowledge acquisition in structured domains. Proceedings of the Tenth Annual Conference of the Cognitive Science Society (pp. 375–383).

Spiro, R. A., Feltovich, P.J., Jacobson, M.J., & Coulson, R.L. (1991). Cognitive flexibility, constructivism, and hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains. Educational Technology, 31(5): 24–33.

********** End of Document **********

IMEJ multimedia team member assigned to this paper: Yue-Ling Wong