IMEJ main Wake Forest University Homepage Search articles Archived volumes Table of Content of this issue

Developing a Strategy for Creating and Assessing Digital Media Curriculum Material
Jennifer Burg, Wake Forest University
Yue-Ling Wong, Wake Forest University
Leah McCoy, Wake Forest University

Abstract
Creating curriculum material in digital media is a challenge, not only because of its many facets, but also because of the tug-of-war between application-oriented instruction versus fundamental concepts. Our work is an attempt to respond to the challenges of digital media curriculum development by dividing the material into three modules with parallel chapters: a primer, an art module, and a computer science module. An additional component of our digital media curriculum development project is an assessment of the pedagogical value of the instructional modules we have created. One of the special challenges in assessing digital media curriculum material is making the connection between concepts and applications. The two traditional methods of assessing student performance -- evaluating either their final projects or their mastery of concepts through tests and quizzes -- fail to associate "knowing" with "doing." This paper describes an assessment experiment recently conducted to help us gain insight into students' thought processes. Students are observed and videotaped in a closed lab designed to help them understand concepts and to help us assess how they relate concepts to activities. Based on our observations, we propose a refinement to our assessment procedures and improvements to our curriculum material.

1. Background 
Digital media is an inherently interdisciplinary area of study with links to computer science, art, film, music, communications, architecture, and design [Burg, Wong, and Strokanova 2004]. Creating appropriately targeted curriculum material in digital media is a challenge, not only because of its many facets, but also because of the tug-of-war between application-oriented instruction versus fundamental concepts. Our work is an attempt to respond to the challenges of digital media curriculum development by dividing the material into three modules: a primer of basic material important to students regardless of the perspective from which they approach digital media; an art module presenting concepts and techniques important to art students; and a computer science module explaining the mathematical, algorithmic, and technical underpinnings of digital media. Each module consists of traditional text-based chapters accompanied by interactive on-line demos, worksheets, and programming exercises as appropriate to the subject. Parallel chapters run through all three modules in the areas of digital imaging, digital audio, digital video, and multimedia programming.

Students who take digital media courses are interested in learning how to do photographic processing, audio/video production, and multimedia programming with the tools and languages of the day. For practical reasons, digital media curriculum needs to include specific application programs -- for example, Photoshop, Premiere, and Flash. However, focusing too narrowly on specific application programs has its drawbacks. One program works differently from another, even within the same medium (e.g. digital imaging or digital audio). In addition, versions of these programs change quickly, sometimes within a matter of months. Our goal is to create material that explains fundamental concepts of lasting utility, and to explain these concepts in a way that facilitates the students' ability to work in application programs and to be adaptable to new applications when they encounter them. For the primer and art module -- which is aimed at students who may have a non-technical or non-mathematical background -- this entails using simple analogies from everyday life, and showing how the concepts are important to their use of digital media tools in a generic explanation. For the computer science module -- which explains the mathematics and algorithms behind the scenes of digital media -- this entails identifying the material that is important to the students' advanced use of their tools. In the case of all three modules, it is important to identify the concepts that students most often misunderstand, and to find better ways to explain these.

An additional component of our digital media curriculum development project is an assessment of the pedagogical value of the material. Meaningful assessment is never easy in educational research, and in some ways we have found this part of our work to be the most difficult. The digital media course we teach is offered only once a year and typically has between 10 and 30 students in it -- a small population on which to base a significant statistical analysis. The small size of our classes also precludes doing comparative studies where one group uses our curriculum material while another uses other sources. In any case, such comparative studies entail treating groups of students unequally and risk disadvantaging one group in favor of another. Comparing student performance from one year to the next is another approach. But this method of assessment is slow-going, and results are difficult to interpret when comparison groups have different backgrounds and skill levels from the outset.

In developing an assessment plan, we are dealing with these problems in two ways. First, we are working with another university with large classes where comparative studies or pre/post testing might yield statistically significant results. Secondly, we are trying to devise a common-sense, practical approach to assessing our own students' performance based on observing what they do and do not understand. This paper describes an experiment recently conducted to help us develop our "in-house" assessment strategy.

2. The Assessment Experiment
Digital media courses involve an integration of concepts and applications, and thus a proper assessment of digital media curriculum material involves evaluating the students' understanding of concepts and their application of concepts in hands-on activities. Thus, one way to assess student performance might to look at the students' finished work in projects assigned in a digital media course -- the images, audio files, videos, or multimedia programs they create. However, it is not easy to determine how well students understand concepts simply by looking at their completed work. Digital media application programs generally have default settings and powerful high-level commands that students can use -- through trial and error -- to produce polished finished products even when they don't understand what the settings mean. Also, there are multiple ways to get the same results, and it is not possible to assess exactly what steps a student went through to accomplish any given result. On the other hand, traditional tests and quizzes might assess a student's understanding of concepts, but they reveal nothing about how the student succeeds with hands-on activities. What is needed is an assessment strategy that evaluates the students' understanding and application of concepts in the context of using digital media tools.

This paper reports on our attempt to develop an appropriate strategy for assessing the pedagogical value of our curriculum material by experimenting with an assessment approach applied in the context of a learning unit on digital audio processing. The goals of this study follow:

  1. To evaluate how well the students could navigate through an unfamiliar application program, understanding the concepts they had learned as these concepts were applied in a hands-on setting.
  2. To note the concepts that still gave students trouble.
  3. To identify concepts that we had not explained in the curriculum material but that were important to their effective use of the application tools.
  4. To evaluate which of the curriculum approaches (text-based, worksheets, on-line demos, in-class lecture) the students found most useful to their understanding.
  5. To apply what we learned in our experiment to developing a general strategy for continued development and assessment of digital media curriculum material.

The assignment upon which this paper is based was given to a digital media class of ten students in the spring 2004 semester. The students' learning objectives for the homework and lab exercise are described below:

  1. To learn basic concepts of digital audio, including sampling rate, aliasing, quantization, quantization error, dithering, frequency analysis, dynamic range, audio compression, and common audio file types.
  2. To gain practical experience with the concepts listed above in the context of a digital audio editing program.

The assignment had eight components -- seven items to be covered by students as homework, and the eighth constituting a closed lab worked on by three groups of three or four students per group. The homework assignments included the following:

The demos on Sound Fundamentals and Audio Dithering were on-line tutorials written in Macromedia Director/Shockwave. The worksheet on Non-Linear Companding was to be done in MatLab.

Text-Based Material:
demo: file (pdf, ~250 KB) Primer Chapter 4: Fundamentals of Digital Audio
demo: file (pdf, ~200 KB) Computer Science Chapter 4: Digital Audio

Pencil-and-Paper Worksheets:
demo: file (pdf, ~27 KB) Digital Audio Sampling Rate, Quantization, and File Size (computer science module)
demo: file (pdf, ~15 KB) Conversion Between Air Pressure Amplitude and dB_SPL (computer science module)

Multimedia Learning Aids and Computer-Based Worksheets:
demo: interactive (Shockwave, ~ 460 KB) Sound Fundamentals (primer)
demo: interactive (Shockwave, ~ 600KB) Audio Dithering (computer science module)
demo: file (pdf, ~76 KB) Non-Linear Companding (computer science module)

The students were given a week and a half to complete the reading and worksheets. They then met for a two-hour closed lab session where they worked in homogeneous groups of three or four students, divided according to their test averages in an attempt to equalize the level of individual participation in each group. In the closed lab, the students completed an exercise using Adobe Audition, an application program with which the students had no previous experience. The exercise required that the students

3. Assessment of Student Learning and Student Reaction to Curriculum Material
Students were asked to rate the various components of the curriculum material according to which one helped most in their understanding of each concept. Students were also asked to indicate if they did not complete one of the components. A component that was not completed by a student received no rating from that student and did not affect the average rating for that component. The ranking of the usefulness of each component is given in the table below. Lower numbers indicate a better rating; i.e., 1 means that the curriculum component was considered most helpful by students, as determined by the average rating. (Averages are not shown in the table.) The table also indicates how many students did not complete each component.

The component most often not completed by the students was the MatLab exercise in Non-Linear Companding. In general, the students found the class lectures most helpful, with the text-based material a close second. With regard to their preference for the text-based material over the on-line material, we find it difficult to draw any conclusions. The on-line demos for the audio unit (Sound Fundamentals and Audio Dithering) are not the best ones among our curriculum material in the sense that they are fairly short and not as interactive as they could be. The exercise on Non-Linear Companding -- also an interactive computer-based exercise -- was carefully constructed and we expected it to be helpful in clarifying the concepts of µ-law encoding both graphically and mathematically. However, half the class failed to complete this exercise, probably because it was the last assignment handed out prior to the closed-lab, and also because the students were required to use MatLab to do the exercise, a program that some of them had not used before. When pressed for time, students omitted doing this exercise since they were not required to turn in their answer sheet for it.

Topic Student Rating Number of students who did NOT complete this component
Sampling and Aliasing
Primer Chapter 4: Fundamentals of Digital Audio 2 0
CS Chapter 4: Digital Audio 3 1
Interactive demo: Sound Fundamentals 4 1
Class lecture 1 0
Filters and Transforms
CS Chapter 4: Digital Audio 2 1
Class Lecture 1 1
Quantizing, Quantization Error, Dithering
Primer Chapter 4: Fundamentals of Digital Audio 1 0
CS Chapter 4: Digital Audio 3 1
Interactive demo: Sound Fundamentals 4 1
Interactive demo: Audio Dithering 5 1
Computer-Based Worksheet: Non-Linear Companding (MatLab) 6 5
Class lecture 2 0
Dynamic Range, Decibels
Primer Chapter 4: Fundamentals of Digital Audio 3 0
CS Chapter 4: Digital Audio 2 1
Interactive demo: Sound Fundamentals 4 1
Conversion between Air Pressure Amplitude and dB_SPL 5 3
Class Lecture 1 0
File Size, File Type, and Compression
Primer Chapter 4: Fundamentals of Digital Audio
3 0
CS Chapter 4: Digital Audio 2 1
Interactive demo: Sound Fundamentals 4 1
Digital Audio Sampling Rate, Quantization, and File Size
5 1
Computer-Based Worksheet: Non-Linear Companding (MatLab) 6 5
Class lecture 1 0

Table 1. Students' rating of how much they learned from each learning unit.

In order to keep a record of the students' work and interactions in the closed lab, we recorded their screen activity with Camtasia and set up video cameras to record their group interactions as well. These videos helped us to pinpoint where students misunderstood concepts or had trouble navigating in the application program. For another example of video-analysis, see [Maor and Knibb, 1999].) Table 2 summarizes our analysis of the students' knowledge and reasoning as evidenced in the videos and their scores on the written answer sheets for the closed lab.

Topic Corresponding Questions in Closed Lab Exercise Group Knowledge (based on group discussion) Reasoning (based on group discussion) Score per section (based on written answers)
Sampling and Aliasing
 
questions 2,3 group 1 4 4 6/6
group 2 4 4 6/6
group 3 4 3 5/6
Filters and Transforms questions 1, 4-6 group 1 3 4 9/9
group 2 2 3 7.5/9
group 3 2 1 5.5/9
Quantizing, Quantization Error, Dithering questions 7-11 group 1 4 4 10/10
group 2 3 3 10/10
group 3 3 2 10/10
Dynamic Range, Decibels questions 12-15 group 1 4 3 6/6
group 2 4 3 6/6
group 3 2 2 5/6
File Size, File Type, Compression questions 16-19 group 1 4 4 4/4
group 2 4 4 4/4
group 3 3 3 4/4

Table 2. Assessment of students' knowledge and reasoning as evidenced by the closed lab worksheet.

In many cases, the closed lab exercise appeared to be successful not only in demonstrating the students' mastery of the concepts learned in the homework, but also in reinforcing these concepts by requiring the students to reason through the questions. The Camtasia and video excerpts below show cases where the lab exercise was successful in directing the students toward sound reasoning and better understanding of the concepts.

The concepts or features of the audio editing program that students had trouble with are listed below. Some are illustrated with Camtasia or video excerpts. (Note that some activities listed above as "successful" are also listed below as causing trouble for students. This is because the groups did not do equally well on all activities and questions).

From these observations, we determined that we needed to revised our curriculum material to include a more detailed explanation of windowing functions for the fast Fourier transform; additional reinforcement of the difference between sampling and quantization in the context of digital audio; a clearer explanation of the implications of dynamic range; and more reinforcement of the interpretation of spectral views. We would like to illustrate these concepts in short interactive tutorials with questions that engage the students' active learning.

The Camtasia and video excerpts:
demo: movie Video clip (QuickTime, ~17 MB): understanding the effects of aliasing.
demo: movie Screen capture (Flash, ~1 MB): successful mastery of applying a notch filter to filter out an aliased sound wave component resulting from downsampling.
demo: movie Video clip (QuickTime, ~8.5 MB): correctly interpreting a spectral view.
demo: movie Screen capture (Flash, ~1.5 MB): getting a better understanding of dynamic range.
demo: movie Video clip (QuickTime, ~10 MB): understanding and hearing benefits of µ-law encoding in reducing file size while preserving dynamic range.
demo: movie Video clip (QuickTime, ~13 MB): understanding how to read the spectral view of a wave form.
demo: movie Video clip (QuickTime, ~27 MB): confusing the effects of sampling rate reduction (aliasing) with the effects of bit depth reduction (quantization error and distortion), and misunderstanding the implications of dynamic range as restricted by bit depth.
demo: movie Video clip (QuickTime, ~5 MB):understanding how dithering works -- i.e., does it "smooth" a sound wave or make it more irregular.

4. Conclusions Related to Future Plans for Assessment
Our conclusions from the experiment are the following:

5. References
Yue-Ling Wong, Jennifer Burg, and Victoria Strokanova, "Digital Media in the Computer Science Curricula," Proceedings of 35th SIGCSE: Technical Symposium on Computer Science Education, Norfolk, Virginia, March 3-7, 2004, pp. 427-430.

Maor, Dorit, and Ken Knibb, "Video Analysis: A Qualitative Tool for Investigating Students' Learning in a Constructivist-Oriented Multimedia in a Science Classroom, http://www.aare.edu.au/99pap/mao99401.htm.

6. Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant No. DUE-0127280 and DUE-0340969.

********** End of Document **********

© 2004 Wake Forest University (from Volume 6, Number 1, of The Interactive Multimedia Electronic Journal of Computer-Enhanced Learning).