Printer-friendly version

Scaffolding Students from Knowing to Applying: Using COR to Teach Observational Research Methods
Carla N. Torgerson, Penn State Erie , The Behrend College
Dawn G. Blasko, Penn State Erie , The Behrend College
Victoria A. Kazmerski, Penn State Erie , The Behrend College
Jacob A. Cornwell, Penn State Erie , The Behrend College

Abstract
This paper discusses Courseware for Observational Research (COR), an interactive multimedia program designed to teach methods of observational research from the initial phase of hypothesis generation through the final stage of data analysis and interpretation. The program uses video with interactive coding activities, a case study, and a laboratory component to teach students key concepts and applications. The program is designed to gradually decrease the level of assistance until students are applying the research methods on their own. The case study, in particular, helps students to apply their learning in a realistic setting. Evaluations have shown that COR enhances student confidence and improves learning.

About the authors...

1. Introduction 
Courseware for Observational Research ( COR ) is an educational multimedia program developed at Penn State Erie, The Behrend College. It is an interactive program designed to teach observational research methods from the initial phase of hypothesis generation, through data analysis and interpretation, and finally to presentation of results in the form of a report. In the current paper we explain the instructional problems we faced and then describe how COR was designed to provide solutions. Multimedia examples are used to describe the components of the program and how it is being used in our classes. Finally, we describe the promising results of preliminary evaluations and our plans for the future.

2. Instructional Need
COR was designed to address serious shortfalls in science education by emphasizing observational research as a tool for scientific inquiry. As acknowledged in the National Science Education Standards, this process of inquiry is “central to science learning” (National Research Council, 1996). Observational research can be used to study behavior in and of itself. Examples of such work are that of Jane Goodall and Dian Fossey (Fossey, 2000), which have been popularized in the films Jane Goodall's Wild Chimpanzees, and Gorillas in the Mist. Systematic observations are also the foundation of experimental methodologies that are heavily used in physical, biological, and social sciences.

Science education has not done a very good job of making science an attractive field for women ( Lynch, 2000). We believe a major factor is that many of the topics that are usually studied when learning the scientific method are less interesting to girls (e.g., soil samples, earthworms, levers and pulleys). By highlighting the more complex and interesting behavioral sciences through the use of videos with topics such as children playing, college students' social behavior, and gorillas feeding, COR may help to enhance women and girls' interest in science.

The scientific method requires the understanding of probability and statistics; however, even in cases where students have taken statistics in a previous course they have difficulty applying what they have learned to their research methods course. Students find the research methods courses difficult and boring, leaving faculty members teaching unhappy and unmotivated students (Blasko, Kazmerski, Corty, & Kallgren, 1998). As a result, research methods and statistics courses often receive lower student evaluations than other courses (Cashin, 1990).

3. COR’s Instructional Model
Our solution was to create an interactive multimedia program to teach observational research concepts. In each lesson a short video is used to present observational settings that are realistic and have been carefully chosen for their relevance to a particular concept. Students are able to play the videos as often as they wish and are encouraged to practice on screen the concept that is being presented. Students then go on to practice these concepts with more independence in a guided case study environment, which allows them to select the coding technique they feel will be best for the setting, report their results, determine interrater reliability, and then present their results. Next, students work with the concepts more independently in a laboratory component that consists of other videos that, again, are excellent examples of various human and animal behaviors. In the lab section, there is very little guidance to help students; however, links are provided throughout the lab that allow them to go back to the lesson material and review the concepts in greater depth. Finally, students can access a library of videos where they can practice their observational research skills with no guidance. Our goal was to design the program to scaffold students from more instructor-directed learning in the lessons to more and more student-directed learning in the case study and then the lab and library. As students gain more confidence, the program gives more choices and provides less guidance.

Despite the stereotype of the lonely scientist, the scientific process is largely collaborative in nature. Typically scientists work in teams and a diversity of skills and learning styles can enhance creativity and efficiency. The hallmarks of good research are that the findings are both reliable (repeatable) and valid (generalizable to new populations). Therefore it is critical to compare the results of one observer to another independent observer. By focusing on this, COR was designed to enhance collaboration. Students work in small teams of two or three to define their research question, develop the best methodology to test it, conduct the observation, and then test their interrater reliability. This type of collaboration can develop teamwork skills and enhances learning by encouraging deeper processing of course materials as students explain concepts to each other (Jones & Carter, 1998; Teasley, 1995).

3.1 Lessons
Students COR consists of 6 lessons. These lessons were initially developed to be used within class time, but can also be used by students out of class for further practice. The first lesson introduces observational research. Subsequent lessons discuss a particular concept of observational research (coding strategies, sampling strategies, assessing interrater reliability, hypothesis testing using Chi-Square, and presenting results in APA format). Each of these subsequent lessons uses short videos to present a situation in which that concept can be applied. (VIDEO 1) The videos are carefully chosen for their relevance to the concept and ability to enhance the teaching of the concept. Using these videos, instructors can discuss issues and immediately demonstrate them for the class. Segments of the video can be replayed to emphasize a particular issue or idea, or to help answer a student question.

Outside of class, the student can also access the lessons for additional practice. The student has the option to watch only the video, to code the video while watching it, or to watch while our fictional expert codes the video. This gives the student great flexibility in learning the observational research concept, including an opportunity to watch the process modeled before he does the coding himself. For example, in learning about coding techniques, the student can choose to work on frequency coding, duration coding, or interval coding. If the student chooses to work on frequency coding, she will watch a video of a gorilla and code the frequency of its eating behavior. The student can start by watching the video to familiarize herself with what will be happening. Then she can watch the expert code the video as a model. If she prefers, she can code the video herself or do this after watching the expert model the technique. After mastering one technique the student can continue on to another, watching another person or animal’s behavior. This sort of variety also makes the program fun and interesting.

Screen capture (~2.5 MB) demonstrates the duration coding lesson in the COR.
Require Shockwave plugin.

A subsequent lesson takes the student’s data and shows how to perform interrater reliability testing. The student is permitted to use the expert’s data or his own. In using his own data he can see how tests of interrater reliability (percent agreement and Cohen’s Kappa) are applied to his data. He then evaluates whether his data have sufficient interrater reliability to continue with data analysis and significance testing.

Figure 1. A subsequent lesson takes the student’s data and shows how to perform interrater reliability testing.

Screen capture movie (AVI, ~1.6 MB) shows a COR lesson that takes the student's data for interrater reliability testing.
Require Camtasia TSCC codec installed.

3.2 Case Study
Case studies are an excellent learning tool as they allow students to apply concepts in an authentic setting (Johnson & Solso, 1978; Alessi & Trollip, 2001). Case studies also enhance transfer of learning (Alessi & Trollip) so it is anticipated that after using the case study in COR, students will be better able to transfer their learning to other settings when doing their own observational research. The case study module in COR was also developed to scaffold the learner from the more teacher-directed focus of the lessons component to the more student-directed focus of the lab.

COR presents a case study in which a fictional young girl, Sarah, may be removed from her afterschool program due to her aggressive behavior towards the other children. Sarah’s teacher claims the child is not aggressive, but a staff member at the care center claims she is. Sarah’s parents have come to you for help in determining whether Sarah really is behaving any more aggressively than her peers or if gender stereotypes or other issues are influencing the staff member at the care center. Students observe Sarah in a play setting and assess if her behavior is more aggressive than her age-matched peers. Students must use two observers and calculate interrater reliability, and then interpret if their results are statistically significant. Afterwards students are presented with the difficult task of determining if Sarah should be removed from the afterschool program. To demonstrate the complexity of such decisions, students then have the opportunity to watch Sarah play in another setting where she acts very differently.

At the beginning of the case study students are introduced to Dr. Wellington, a fictional expert in observational research who is there to help them through the case study. She guides students through the case by asking questions about things such as how they should observe Sarah, how they will design their coding sheets, what their level of interrater reliability is and if that is acceptable, if their results are statistically significant, and what their results tell them. At every step, students are given feedback from Dr. Wellington that helps them understand what they just did or how it impacts the next step. Dr. Wellington also tells students when they give incorrect answers, but she doesn’t just tell them the answer; she gives hints, tips, and other leading feedback to make students think more deeply about the question they answered and try again. Through answering these questions, the student is led through the entire observational research process.

We believe this “guide on the side” approach is an invaluable way for students to learn. They are able to think about what it is they are doing and are given feedback at every step. This prevents a student from making a mistake early on that causes the rest of the assignment and his practice with the concepts to be incorrect. Students are given lots of help, but in a way that makes them think about exactly what it is they are doing, as well as why and how it applies in the given context of the case study. This is something a textbook simply cannot do. The case study also concludes by asking students to write a report that outlines their results and conclusions regarding Sarah. This can be graded by the instructor, allowing the case study to be used as a class assignment.

Figure 2. The case study starts by introducing the student to the scenario and characters. Clicking on a picture with a sound icon will allow the student to hear that character speak.

Screen capture movie (AVI, ~ 4.6 MB) demonstrates the learning process through the case study in the COR.
Require Camtasia TSCC codec installed.

3.3 Laboratory
The laboratory component is designed to allow students to practice the observational research methods with very little guidance. After completing the lessons with their instructor and the case study in collaboration with their virtual guide Dr. Wellington and a student partner, the laboratory serves as a way for students to apply their growing knowledge. This part of the program can be used by the instructor as a full laboratory assignment and assessed to determine mastery of the critical concepts. Alternatively it can be used as preparation for actual field observations. In the laboratory module, the student is given a choice of seven video subjects to observe (tropical fish, aggressive driving, children at quiet play, spider monkey social interaction, movement of boats in a channel, feeding birds, and hand gestures of a couple discussing the floor plan of their new home). Students can preview a short clip of each of the videos before deciding which one to use. Working collaboratively, pairs of students are prompted through the steps of developing a hypothesis and setting up an observational coding scheme. After the coding scheme is finalized, students watch and code behaviors in the video and then assess their interrater reliability and test their hypothesis. Because little guidance is given in the lab component, links are provided at each step of the lab exercise to allow students to go back to the lesson material and review the concept in greater depth. This allows the student to get extra help if needed.

Evaluation of the laboratory module was very positive. Students and instructors felt that the strength of the laboratory was its step-by-step progression through the stages of the research process. They thought the quality of the videos was good, but would have liked even more videos with better quality; in response we have just completed expanding the video library. Instructors have also expressed interest in the addition of a student assessment that could be saved and printed or e-mailed to the instructor.

(~500 KB) Screen capture shows the Laboratory component of the COR.
Require Shockwave plugin.

3.4 Library
The library provides a collection of all videos used in the program. This part of COR allows the student to watch the video and apply the research methods without any guidance. These can be used for class assignments or simply for students to do further practice. In the library there is also background information about the subjects of each video and the context in which they were filmed as well as audio interviews with caretakers of some of the animals. Instructors find the library useful because it gives them the flexibility to use any video to discuss any concept. For example, we often use different situations to learn to develop precise operational definitions (e.g., how is feeding behavior similar or different in gorillas, flamingos, and college students).

3.5 Glossary
COR also has a glossary section that provides definitions of difficult terms. Students can browse through the glossary or search for specific terms. The search utility is flexible enough to allow for common misspellings. The instructor is also able to add terms to the glossary and/or modify definitions at any time.

4. Programming Environment and Use of Technology
COR version 2 has been developed using Macromedia® Director®. Director® is an excellent tool as it allows for integration of videos, a high level of interactivity, and storing student data. Tracking student input is especially important within the case study where students are responding to questions and getting appropriate feedback along the way. Students make numerous choices which are reflected on the subsequent pages and influence how they deal with the rest of the case study. This is particularly important in the sections where students are evaluating interrater reliability and statistical significance as it is their own data that is being worked with on the screen.

Integration of videos is a key to COR’s ability to assist students in enhancing their observational research skills. With Director® we were able to create “Dr. Wellington’s coding” where a model coding sheet is filled in while the video runs. By tracking the timing of the video, we are able to show data being written on the computer coding sheet as if the expert were coding the video. This creates a very powerful learning tool for students.

We also capitalized on the ability of Director® to read external files to create a program that can be easily customized by the instructor. Instructors are able to provide different coding for a particular video, either to modify the way the computer codes the video or to have the computer code the video for a different behavior. In much the same way, the instructor is also able to modify the glossary by adding more terms or editing the definitions provided.

5. Use and Evaluation
Extensive evaluation of the lesson module was conducted with COR Version 1. One class of introductory psychology was taught observational research using COR and the other with traditional lecture methods supplemented with examples on VCR. An instructor, who never used COR before, taught both classes. Students then worked in pairs to design and conduct an out of class observation. Multiple choice questions were included on the exam to assess knowledge. The critical assessment consisted of a complete APA style lab report completed collaboratively by each team. Graders assessed these papers while blind to the students’ identity and training status (COR vs traditional). The results showed that the groups did not differ on scores of the 10 multiple choice exam questions (t(82) = 0.51, p > .05). However, when compared to the class taught with traditional methods, the COR instructed students scored significantly higher in their hands-on laboratory assignment (t(81) = 2.80, p < .01). Additionally, the COR instructed class rated the usefulness of conducting research more highly than the group taught without COR, t(71) = 2.17, p < .05 (Kazmerski & Blasko, 1999).

Although additional material has also been added to the lessons, the major innovation in COR Version 2 is the interactive case study module. In the spring 2003 semester, COR 2 was used for the first time in a Basic Research Methods class at Penn State Erie. Because we had extensively evaluated the lessons as described above, here we focused our attention on the new case study. The program was evaluated in several stages. Students read material on nonexperimental designs (Bordens & Abbott, 2002) and completed an on-line quiz before coming to class. The instructor then presented the lessons component of COR during five hours of class time. Students took a pre test based on their knowledge to date and were then given a list of topics and asked to rate their confidence in actually performing the task. Students then used the case study at individual laptop computers in class over five hours. The class went to the zoo to complete an observation in a two-person team and wrote up the results of their observational study in an APA style laboratory report. They then rated their confidence in the same topics of observational research as before. All students completed a post test on the concepts as part of their course exam. In the final step all students completed a usability survey about COR and this information was used to make improvements to the program.

Students rated 10 topics, such as conducting Chi Square analysis and assessing interrater reliability, on a scale of 1 (not at all confident) to 10 (extremely confident). The results of a series of paired-samples t-tests showed significant increases in confidence across all ten topics after using the case study. Shown above is the composite score, illustrating how mean confidence ratings of 5.4 after completing the lessons significantly increased to 8.4 after completing the case study, t(21) = 11.9, p < .0001. Although less dramatic, there was also a significant increase in the test scores of students after completing the case study, t(22) = 4.7, p < .0001. This was despite the fact that students were asked to calculate Chi Square without the benefit of formulas or computers.

The results of the usability analysis showed that students felt that both the lessons and the case study were attractive, interesting, and easy to use. Students had difficulty creating and using data files in the case study and we’ve modified the program to alleviate this. Many students spent considerable time creating screen captures of the COR lessons which they used when writing their paper and studying for the test. Enhancing the print features of COR is on our list of desired improvements. Overall, students felt strongly that COR was a valuable addition to the class and that they now understood the application of observational techniques in much more depth.

6. Summary and Future Directions
In the fall of 2003 we will be conducting extensive evaluations of the completed COR 2 program. At this point COR Version 2 is ready for complete testing at other schools and with other populations. Currently we have completed a booklet of handouts that the instructor may photocopy for his students and we are also working on books to accompany the program, including a manual for students (explaining the concepts in greater depth) and one for teachers (explaining how to use COR most effectively in the classroom). We have used portions of COR with younger students in summer science camps and would like to spend more time testing and modifying the program for a younger audience.

We would also like to examine the feasibility of delivering portions of COR over the web. Although there remain significant drawbacks due to bandwidth limitations, major strides have been made in recent years in delivering more interactive programs and higher quality video over the internet.

7. Availability
We are looking for partners to help test and evaluate COR Version 2. If you are interested in learning more about COR or how you can use it with your students, please contact Dawn Blasko at dgb6@psu.edu or visit our website at http://www.cor.psu-erie.bd.psu.edu/cor/ .

An external link to the authors' website.

8. References 
Alessi, S. M. & Trollip, S. R. (2001). Multimedia for learning: Methods and development (3rd Ed.). Boston: Allyn and Bacon.

Blasko, D. G., Kazmerski, V., Corty, E., & Kallgren, C. (1998). Courseware for Observational Research (COR): A new approach to teaching naturalistic observation, Behavioral Research Methods and Computing. Vol. 30(2): 217-222.

Bordens, K. S. & Abbott, B. B. (2002). Research design and methods: A process approach (5th Ed.). Boston: McGraw-Hill.

Cashin, W. (1990). Students do rate different academic fields differently. In M. Theall & J. Franklin (Eds.), Students ratings of instruction: Issues for improving practice. San Francisco: Jossey-Bass, Inc.

Fossey, D. (2000). Gorillas in the mist. Boston: Mariner Books.

Johnson, H. & Solso, R. (1978). An introduction to experimental design in psychology: A case study approach (2nd Ed.). NY: Harper & Row Publishers.

Jones, M. G., & Carter, G. (1998). Small groups and shared constructions. In J.J. Mintzes, J.H. Wandersee, & J.D. Novak (Eds.) Teaching science for understanding: A constructivist view (pp. 261-279). San Diego: Academic Press.

Kazmerski, V. A. & Blasko, D. G. (1999). Teaching observational research in introductory psychology: Computerized and lecture-based methods. Teaching of Psychology. Vol 26(4): 295-298.

Lynch, S. (2000). Equity and science education reform. Mahwah, NJ: L. Erlbaum Associates.

National Research Council (1996). National Science Education Standards. Washington DC: National Academy of Sciences Press. Available on-line at: http://www.nap.edu/readingroom/books/nses/html/ .

National Science Foundation (1999). Women, minorities, and persons with disabilities in science and engineering: 1998. Arlington, VA. (NSF 99-338).

Teasley, S. D. (1995). The role of talking in children’s peer collaborations. Developmental Psychology, 3, 207-220.

********** End of Document **********

IMEJ multimedia team member assigned to this paper	Yue-Ling Wong

© 2003 Wake Forest University (from Volume 5, Number 2, of The Interactive Multimedia Electronic Journal of Computer-Enhanced Learning).