Multimodal Rhetoric

Gunther Kress wrote in his paper, Multimodality, Multimedia, and Genre, that “the materiality of the different modes—sound for speech, light for image, body for dance—means that not everything can be realized in every mode with equal facility…meanings, in the broad sense, can be realized in any mode, but that when they are, they are realized in mode-specific articulations” (39). As I hope you have inferred from my discussion on previous pages thus far, text and picture each have different rhetorical strengths and weaknesses. By combining these two elements into a multimodal presentation, a rhetorician could benefit from the complementary effects of text and picture, and his or her overall argument would be strengthened. In Reading Images: The Grammar of Visual Design, Kress and co-author Theo van Leeuwen stated, “We take the view that language and visual communication can both be used to realize the ‘same’ fundamental systems of meaning that constitute our cultures, but that each does so by means of its own specific forms, does so differently, and independently” (19). I agree with their statement if it refers only to distinct mono-modes; if however, they also meant to include the different mono-modal aspects in a multimodal piece, then I take issue with the phrase, “and independently.” I believe that multimodal work is at the peak of rhetorical effectiveness when single modes such text and picture depend on each other for the realization of their meaning.

 

Anne Wysocki wrote a piece titled, Impossibly distinct: On form/content and word/image in two pieces of computer-based interactive multimedia, in which she explored the interaction between words and image, and the rhetorical effect of multimodality. Her method included analyzing two interactive CDs about art collections and artists. Both CDs appeared to contain such similar content that “you might still on first pass think the two CDs were both encouraging you toward constructing similar relationships with art and artists” (Wysocki 3). However, after using both CDs extensively, Wysocki concluded the overall messages one received from the programs were quite different.

The Barnes CD, through its visual presentation of artists’ biographies, is arguing that artists are not worth much attention or differentiation; instead, what is important is the art, the paintings by themselves, as collected things.

The Maeght CD, on another hand, gives equal visual weight to artists, to artists’ lives, and to artwork—but is careful to help us see distinctions between artists even as it shows us how those artists shared places and times (Wysocki 16).

According to Wysocki, the differences of assertion presented by the CDs were due to differences in visual presentation (16). She contended, “I do not think it is possible to make a clean distinction in these CDs between design and information” (24). Her statement echoes the thoughts of other “multimodality investigators” such as David McCandless, who specializes in arranging complex datasets into visually-informative diagrams.

 

One can see in McCandless’ designs that image, text, and information are all intricately linked—in fact, the design is just as “informative” as the information itself. At one point in the video, McCandless mentioned the “language of the eye” and the “language of the mind,” and how, through the combination of these two languages, we can “alter our perspective and change our views.” He emphasized how data visualization can change people’s mindsets and ultimately, their behavior. The result, he proposed, of changed behavior is the solution to problems present in society, economics, politics, etc. “Design is about solving problems and providing elegant solutions,” declared McCandless.

McCandless was not the only person concerned with the possibilities and horizons of data visualization. Jane McGonigal proposed that people could solve world problems through video games. She suggested that gaming dynamics sharpen people’s problem-solving abilities, and if game designers created a virtual world to emulate the real world, gamers would eventually discover solutions to real-world problems through game role-playing. Although I disagree with some of her blanket generalizations (e.g. “gamers are super-empowered hopeful individuals”), I find her theory fascinating:
 

 

Similarly to McCandless’ talk, McGonigal’s speech deals a lot with changing people’s perspective due to data visualization and interaction. Drawing off her overall presentation, we might even say that the accurate perception of information involves the participation of more than one sensory system. Despite the usefulness of multi-sensory, multimodality, it can also present risks when it causes our senses to conflict. A basic illustration of this is demonstrated in the video below using the McGurk Effect:
 

 

Image is powerful, especially when functioning rhetorically in multimodal presentations. Our world is comprised of a system of networks, and people can interact with each other now more than ever before; in view of this, I think video is one of the most effective multimodal rhetoric tool we have available to us. Not only can film combine words and image, but it also involves our other senses such as hearing. Because video incorporates multiple modes and sensory aspects, people naturally become more involved in interpreting the message—and greater audience participation can lead to greater rhetorical pull. Video can be rhetorically effective on an individual basis as well as in terms of the number of people persuaded. When published on the internet, video—and its messages—can reach an audience of millions. Mary Hocks wrote, “Writers now engage in what Porter calls ‘internetworked writing’—writing that involves the intertwining of production, interaction, and publication in the online classroom of professional workspace as well as advocating for one’s online audiences” (631). Writing will always have an important place in human society, but its role as chief rhetoric will eventually—I believe—be usurped by video.