Glossary

Printer-friendly Version

Student Explanations and Patterns of Use In a Hypermedia Learning Environment
Tricia Jones, University of Michigan
Gail P. Baxter, Educational Testing Service

Abstract
"Seeing Through Chemistry" is a multi-faceted hypermedia environment designed to support student learning in an introductory college chemistry course. Design features such as content coverage, inquiry questions, and feedback provide students opportunities to make connections among concepts as they develop an understanding of chemical phenomena. In this paper, we consider the impact of these design features on student explanations (proxy for student learning) for the inquiry questions in two of the six instructional modules and their influence on students’ patterns of use. Results indicate that feedback on answer correctness and provision of "model" explanations led students to revise their explanations. Nevertheless the overall quality of explanations was generally low for all inquiry questions; they did not seem to depend on the content coverage. Implications for hypermedia learning environment design are discussed.

1. Introduction
Learning is a process of constructing new knowledge on the basis of current knowledge. As learning occurs, increasingly well-structured and qualitatively different organizations of knowledge develop. Structured knowledge is not just a consequence of the amount of information received, but reflects exposure to an environment for learning where there are opportunities for problem solving, interpreting, working in unfamiliar environments requiring transfer, and making connections between relevant concepts. The inference to be made for the design of effective learning environments is that how you learn is as important as what you learn.

Hypermedia systems may be particularly appropriate as learning environments because the associative knowledge structures of a topic or subject matter are made explicit. However, in creating hypermedia systems, attention must also be given to design features that can support students in making appropriate connections in their own developing knowledge structures (e.g., McKendree et al 1995). Activities that will emphasize the key concepts of a domain and engage the students in problem solving and reflection are especially important in this regard. Examples include guided tours highlighting concepts in a domain and showing how they are related; multiple paths through materials that encourage consideration of the same concepts in differing contexts; questions that promote reflection on the relationships among concepts; and opportunities to generate and revise explanations of key concepts as understanding develops. The inclusion of these and other design features in hypermedia learning environments may have implications for the ways in which the software is used and for the quality of learning that results.

About the authors...

In this paper, we consider the impact of various design features on student learning and patterns of use in a multi-faceted hypermedia environment, "Seeing Through Chemistry" [STC]. The theoretical and empirical basis for designing the STC learning environment (Dershimer et al. 1992) stems from psychological principles of learning such as cognitive flexibility theory (Spiro et al. 1988), supplemented by research on difficulties students have learning chemistry (Lythcott 1990; McDermott 1988), and supported by human-computer interaction guidelines. Cognitive Flexibility Theory posits that individuals learn most effectively when they "experience" concepts and ideas from a variety of perspectives. By revisiting content from different perspectives, the individual develops a rich understanding of a concept and its relationship to other concepts. This deep understanding supports thinking, reasoning, and inference making or what might be termed the flexible use of knowledge (Spiro et al. 1988).

Cognitive flexibility theory does not presuppose a computer-based presentation, but hypermedia and multimedia environments are well suited to the provision of learning opportunities that focus on knowledge construction and use. For example, STC integrates video, animations, and experimental simulations with text and pictures onto multimedia content cards to facilitate the development of students’ understanding of topics in an introductory college chemistry course (Dershimer & Rasmussen 1990). One goal of the STC software is to enrich students’ experiential background and allow them to "see" scientific phenomena. Some students lack laboratory experiences and may have never seen a violent acid reaction or light bending through a prism. Without a common experience base, they may not understand topics discussed in introductory college chemistry lectures. Consequently, STC provides students a visual representation of chemical phenomenon through a rich variety of multimedia components (still pictures, animations, video, simulations, and focus questions). Four interaction modes (self-guided inquiry, directed inquiry, structured inquiry, and guided inquiry), distinguished by the level of structure and support, focus students’ attention on the relationship among topics in each of the six modules and allow for multiple paths through the material to support individual interest and use.

An interactive demo (80 KB) of a part of the Acids & Bases module.
(Shockwave plugin required)

Throughout STC, students make decisions on how the software is to be used – whether and when to read the content cards or view supplemental media; which of the four navigation modes to use; and whether and when to revise their explanations. How students respond to features of the software is referred to as patterns of use. Patterns of use are particularly relevant in hypermedia learning environments, where the student has the flexibility to make choices in interacting with the software, because these choices may impact the extent of learning. We cannot assume that all students have seen the same material.

In what follows, we consider two STC modules and examine the quality of student explanations provided in response to the inquiry questions. We chose these modules because they vary with respect to the opportunities for criss-crossing content cards when responding to inquiry questions. We expect that students will learn more (provide higher quality explanations) from instructional modules that provide more opportunities for criss-crossing. Our analysis of student answers and explanations to the inquiry questions provides a general measure of the impact of this design feature on student learning. Next, we examined student patterns of use and their relationship to students’ explanations to inquiry questions (used as a proxy for student learning or understanding) to elaborate on these general findings. We conclude with recommendations for software designers with regard both to interaction models and content design.

An interactive demo (21 KB) on an inquiry question.
(Shockwave plugin required)

Screenshots showing a concept map and the "criss-crossing" concepts for: the Acids & Bases module (20 KB) and the Solubility module (24 KB). (Click on the screenshot to toggle the concept map between with and without the illustration of "criss-crossing".)

2. Methods
2.1. Participants and Context
This study is part of a larger research project examining the use of Seeing Through Chemistry in an introductory chemistry course at a large Midwestern research university. Use of STC was required, but participation in the study was not: 201 of the 450 students enrolled in this multi-section lecture course volunteered to participate. Of the 201 participants, 180 completed the modules of interest in this study.

The semester these data were collected, all students used the software regularly, for graded assignments. Students answered the inquiry questions for five of the six modules in STC and turned in the printed version of their responses. The printout shows the answer choice and the explanation (initial and final). Teaching assistants checked the assignments for completeness. Students received up to 6 points on each assignment for a maximum possible of 30 points (5% of the overall course grade).

2.2. Study Design

Each of the six modules in STC, although designed with the same fundamental instructional goal, varies in terms of the content, the amount of coverage given to a specific topic within a module, and the opportunities for students to examine relationships among topics in a module. These module characteristics may influence how students interact with the material and the level of learning achieved. In order to examine that question, we set out to identify two modules that varied widely in coverage. The modules selected for the study are shown in Figure 1. Acids & Bases has 6 inquiry questions, and was the fourth module completed in the semester. Solubility has 10 questions, and was the fifth and final module completed.

For each of the STC modules, we constructed visualizations that identify the set of topics suggested for each inquiry question. The goal of creating these visualizations was to get a picture of the opportunities the software provides for "criss-crossing" or making connections among topics (i.e., topic coverage). We expect that more opportunities for examining relationships among topics (i.e., criss-crossing) will result in higher levels of understanding (i.e., better quality explanations). Ideally,

(a) all topics would be visited by a student prior to completing a particular module (minimally complete topic coverage),

(b) each topic would be visited for more than one inquiry question (repeat topic coverage), and

(c) a number of the same topics would be suggested for various inquiry questions (overlap of topic coverage).

Reading across the rows of Table 1, bullets (•) indicate when a topic (one or more content cards) is suggested for each inquiry question in a module. For example, in the Acids & Bases module, STC suggests topics 2-4, 4-1, 4-2, and 4-3 are relevant for answering inquiry question 1(topics are named in the software, but names have been omitted for the purposes of this discussion).

2.3 Measures
2.3.1 Log Files and Inquiry Question Responses
STC automatically captures the sequence and duration of choices made by the student. This information is recorded in files known as log files, one per student. Each log file captures completely a student’s use of STC throughout the semester including navigation choices, content selection, answers to inquiry questions, and revisions to explanations. This information can be used to identify patterns of use (what students do), characterize the quality of student explanations to inquiry questions (how well they perform), and thus examine the relationship between patterns of use and performance.

2.3.2 Patterns of Use
A variety of software tools derive patterns of use from the participants’ log files. Moves are extracted from the log files, and characterized by type (e.g., view question (q), view content card (c), edit response (r)). The sequence of moves is combined and then abstracted to form a "pattern of use" and a corresponding label is assigned to each inquiry question (Jones & Jones 1997).

Questions only (qr): read inquiry question, respond

Cards first (cqr): read content cards, read inquiry question, respond

Question guided (qcr): read inquiry question, read content cards, respond

Cards before & within (cqcr): read content cards, read inquiry question, read content cards, respond

2.3.3 Explanation Quality
STC provides feedback on the correctness of student responses to inquiry questions, but it does not evaluate the quality of students’ explanations. A post hoc scoring scheme was designed to focus on the completeness, coherence, and accuracy of student explanations because research in cognitive psychology suggests that the quality of one’s explanations are indicative of the level of one’s understanding (e.g., Chi et al. 1989; Chi, Feltovich, & Glaser 1981). Initial and final (revised) explanations were read and assigned scores from 0 to 4 as follows: 0–Inadequate; 1–Fragmented; 2–Partial, 3–Good, and 4–Elaborated (Table 1). For example, a student who mentions two isolated facts but doesn’t relate them to each other or to the question might be given a level 1 score (Fragmented). A student who mentions all the relevant concepts, including an explanation of "mechanism," and who adds ideas not found in the model response would receive a level 4 (Elaborated). The first author scored both initial and final explanations to the inquiry questions. A second rater scored a 25% random sample. Inter-rater reliability was 0.87 and 0.79 for Acids & Bases and Solubility, respectively.

3. Results

The design of instructional software determines which teaching and learning activities can be supported by the learning environment, and those activities in turn greatly influence the learning that occurs, through the nature and quality of the interaction they support. In research concerning instructional learning environments, it is important to examine both whether the activities occur and the expected outcomes result (Ehrmann 1998).

3.1 Answer Choices and Explanation Quality
For each inquiry question students were required to select an answer and generate an explanation. Answers were scored right or wrong. In general, the answer choices to inquiry questions were relatively easy for students (see Figure 2). In Acids & Bases, 80% or more of the students made the correct choice on 4 of the 6 inquiry questions. The most difficult question in this module was Question 4, with only 20% of the students selecting the correct choice. Similarly, the answer choices in the Solubility module were relatively easy. 55% or more of the students answered all of the Solubility questions correctly. 80% or more of the students answered 7 of the 10 solubility questions correctly. Furthermore, the majority of students answered more than half the questions correctly in each of the modules (i.e., 81% and 96% for Acids & Bases, and Solubility, respectively).

In both modules we see that students generally select the correct answer, but they do not provide high quality explanations for their choices. Moreover, students do revise their explanations and the change in the quality of explanations is statistically significant but the magnitude of the difference is small and the result is still generally poor. Furthermore, contrary to expectations, there were no obvious differences between the two modules in overall quality. Therefore, we investigated if differences were related to student patterns of use. We characterized the various patterns of use, the consistency with which students use them within and between modules, and the relationship to explanation quality.

Figure 2. Base Patterns distribution, by question.

Glossary: To see the explanation of the abbreviations.

In both modules, the qcr pattern is used by almost half the students on each question. Although students are allowed to answer the questions in any order, many of them proceed in numeric order starting with question 1. Notice then that, as the module progresses, students tend to refer to cards between the question and the response (e.g. the size of the qcr group increases while qr decreases, and likewise for cqcr and cqr). This is contrary to expectations: we expected students would visit cards for earlier questions, and then not refer to them again.

In order to examine effectiveness of strategies, we examined mean scores within the base pattern groups. Initial explanation quality score means and standard deviations are shown in Table 2. (Only significant results are shown.) On each of the 16 questions, we conducted a within-subjects (base pattern group: qr vs. cqr vs. qcr vs. cqcr) ANOVA. Base pattern group effects were significant for 5 of the 16 questions. For 4 of these 5, significant group mean differences were found. For two questions (AB 1, Sol 7), the qr group had lower mean scores than all other groups. For a third question (Sol 10), the qr group had lower scores than the two cards-first groups (cqcr and cqr). Consistent with our expectations, these results suggest that patterns of use are related to performance. Students who read cards (either before the question or between the question and response) generate better quality explanations than students who don’t read cards prior to writing their response.

Table 3. Initial explanation quality score by base pattern group.

glossary: To see the explanation of the abbreviations.

Students revise, but at a different level depending on the question (see Figure 4). The percentages range from 20% to 90%, with about 40% to 60% on most questions. It is reasonable to expect students with incorrect choices (the solid blue bars) to revise, since if their choice was wrong it is highly likely their explanation is also wrong. However, even students with correct choices sometimes revise, as can be seen by comparing the size of the first (incorrect answers) and second (revision rates) bars for each question in Figure 4.

In both modules, there is one question where improvement is much less common. For Solubility question 3, only 42% of the revisers received a higher score. In contrast, 72% or more of the revisers improved their scores for all other questions in that module. Interestingly, Solubility question 3 did not have a model response – this perhaps suggests that many students rely on the model response in their revisions. In Acids & Bases, 43% of the revisers on question 5 do not see an improvement. The reason is not immediately obvious, but it might be that students were unable to pick out the crucial concepts in the model response and instead latched onto surface differences.

We wondered if students compare their explanation to the model response. As it turns out, one of the inquiry questions did not have a model response. An examination of responses to Solubility question 3 shows that the question was slightly more difficult than average; 80% of the students answered correctly. While all students with incorrect answers revised, only 20% of the students with correct answers revised. This is the lowest percentage of all the questions. Again, this suggests a reliance on the model response when revising answers.

According to the STC designers, revision should be a key process when students answer the inquiry questions (Dershimer et al. 1992). Substantial revision in the sense of improved explanations is not taking place. Why not? We see that students revise when the inquiry question choice is hard (when they choose the wrong answer). Perhaps the questions need to be harder or trickier. But also, the nature of the interaction – merely expecting students to revise – does not seem to encourage engagement in the content. Students need feedback on the quality of their explanations. A different response model can be seen in the Evolutionary Thematic Explorer (Jacobson 1997). In that system, students choose statements from a list and then have their answers critiqued with respect to two major theories of evolution. Experimental results show learning occurred. Perhaps similar activities here would help: students could assemble an explanation from sets of statements that either capture key concepts or highlight common misconceptions.

5. References 
Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., & Glaser, R. (1989). Self-explanation: How students study and use examples in learning to solve problems. Cognitive Science, 13(1):145–182.

Chi, M.T.H., Feltovich, P.J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science. 5(2):121–152.

Chi, M.T.H., & van Lehn, K.A. (1991). The content of physics self-explanations. The Journal of the Learning Sciences, 1(1):69–105.

Dershimer, R. C., & Rasmussen, P. G. (1990). Seeing Through Chemistry [Macintosh Multimedia Software]. Ann Arbor: Office of Instructional Technology, University of Michigan.

Dershimer, R. C., Wurman, P., Nowak, M. A., Saunders, E., Berger, C. F., & Rasmussen, P. G. (1992). Designing instructional hypermedia to facilitate concept development (unpublished manuscript, available from author).

Dershimer, R. C. (1996). Private communication.

Ehrmann, S. C. (1998). Studying teaching, learning and technology: A tool kit from the Flashlight Program. In Stephen C. Ehrmann and Robin Etter Zuñiga, The Flashlight Evaluation handbook (1.0). Washington, DC: The TLT Group. Also available online at http://www.tltgroup.org/resources/fstudtool.html (6 August 1999).

Jacobson, M.J. (1991). Knowledge acquisition, cognitive flexibility, and the instructional applications of hypertext: A comparison of contrasting designs for computer-enhanced learning environments. Unpublished doctoral dissertation, University of Illinois.

Jacobson, M.J. & Archodidou, A. (1997). Case- and problem-based hypermedia learning environments and conceptual change: The evolution of understandings about evolution. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.

Jones, T., Berger, C. F., & Magnusson, S. (1996). The pursuit of knowledge: Interviews and log files in hypermedia research. Educational Multimedia and Hypermedia Annual, 342–347. Charlottesville: AACE.

Jones, T., & Jones, M. C. (1997). MacSQEAL: A Tool for Exploration of Hypermedia Log File Sequences. Educational Multimedia and Hypermedia Annual, 539–544. Charlottesville: AACE.

Lythcott. (1990). Concept learning vs. problem solving: Revisited. Journal of Chemical Education, 64(6): 510–512.

McDermott, D. P. (1988). Chemistry and the framework of learning. Journal of Chemical Education, 65(6):539–540.

McKendree, J., Reader, W., & Hammond, N. (1995). The "Homeopathic Fallacy" in learning from hypertext. Interactions, 2(3):74–82.

Spiro, R. A., Coulson, R.L., & Feltovich, P.J. (1988). Cognitive flexibility theory: Advanced knowledge acquisition in structured domains. Proceedings of the Tenth Annual Conference of the Cognitive Science Society (pp. 375–383).

Spiro, R. A., Feltovich, P.J., Jacobson, M.J., & Coulson, R.L. (1991). Cognitive flexibility, constructivism, and hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains. Educational Technology, 31(5): 24–33.

********** End of Document **********