Introduction: Multimodality in argumentation

2.1 Multimodal discourse studies: Some interests and issues

Barthes witnessed and pioneered what Mitchell (1994) named a “pictorial turn.” This turn corresponds not only to a growing employment of the visual and the multimodal in everyday and professional communication but also to a growing interest in visual and multimodal artifacts by different disciplines.

Visual and multimodal communication has always existed in both oral and written communication. Oral interactions are multimodal formats combining verbal elements together with intonation, gestures, gaze, and posture. Even written medieval documents were full of illustrations, as Bearden (2012) shows in his work on the Old English Illustrated Hexateuch, an illuminated manuscript from the eleventh century (see also Mitchell 2005; Jewitt 2009: 3). Although we cannot claim that the existence multimodality “lies in the changing of world communication, the rise in digital technologies, and the globalization that results from the dialectical interplay between the two” (Bearden 2012: 2), we cannot even claim that there has been no increase in multimodality. As Bateman (2016) points out:

The use of written documents across all social niches and genres continues to grow rapidly within the vast majority of modern-day cultures. Moreover, […] the combination of verbal language and other modes of expression, including all kinds of graphical, pictorial, typographical and layouting material, is now again sharply on the rise. The previously established restrictions of such combinations to particular genres, themselves sometimes marginalized as in comics or newspapers exhibiting more extravagant layout, are loosening considerably. There is also an accompanying rise in the cultural acceptance of mixed-modality documents as forms of artistic expression, as evidenced in the increasing consideration and awareness of graphic novels, sequential art and comics in both academic and public discourse. Even document types which have traditionally permitted the use of ‘illustration’, such as school textbooks, are making far more use of the technological advances available for presenting material of different kinds than previously. This expansion is driven both by technological possibilities and consumer demand. (Bateman 2016: 309–310)

Technologies enable professional communicators to produce multimodal artefacts with more ease. Annual reports by public companies, newspapers, and other media documents, for example, are designed with an extensive integration of different semiotic means for an audience that is more and more used to this kind of communication. This audience exploits the same technologies to create everyday multimodal communication to be, for instance, shared on social media (think of the wide production of memes and YouTube videos to be shared on Facebook, Twitter or WhatsApp).

The “ascendance” of multimodal communication gave rise to an “increasing interest among academics, professionals, and students in the role of image, gesture, gaze, and posture, and the use of space in representation and communication – in other words, multimodality. “Comments on the multimodal character of communication, texts, and media are increasingly commonplace across a range of disciplines (e.g., anthropology, education, design, linguistics, media and culture studies, musicology, sociology)” (Jewitt 2009: 1). In some cases, the interest in multimodality became a struggle between the verbal and the nonverbal, a struggle said to be often won by the nonverbal. Jackob et al. (2016) go through the most popular examples of studies showing the dominant power of nonverbal cues in persuasive communication. They notice that these are psychological experiments taking place in laboratory setting with controlled variants rather than observations of people interacting in real contexts. It is thus difficult to verify how the findings can be generalized to the real situations.

In the 1990s, many years after Barthes’s first study on the “rhetoric of the image” and his proposal for some categories of image-text relationship, studies within multimodal discourse analysis started focusing again on the text and image divide. Many taxonomies of the possible relationships between the two have been developed (see Bateman 2014 for a complete overview of these taxonomies). However, “analytical methods for handling” the “orchestration [of multimodal documents] are few and far between” (Bateman 2008: 1) ^[1] and the majority of image-text analyses remain actually descriptive (Bateman 2011: 18). The limitation of a descriptive approach is highlighted by Forceville as well because:

these descriptions seldom result in non-trivial explanations why the texts convey what they supposedly do convey, let alone in the formulation of – however tentative – patterns or generalizations. By and by these long descriptions become a real chore to read, and often do not exceed the level of truisms … Bluntly speaking, the authorial descriptions procure insufficient insights that a moderately attentive viewer-reader of the case-study under discussion had not already grasped himself. (Forceville 2007: 1236)

This descriptive approach suffers from the idea that “the ability to deal with multimodal documents [is] acquired implicitly – at least by successful learners” (Bateman 2008: 7). However, education and literacy cannot “afford to continue relying on implicit learning by ‘osmosis’ for multimodality” (Bateman 2008: 7) and cannot be thought “in isolation from a vast array of social, technological, and economic factors” (Kress 2003: 1). It would be more productive, in Bateman’s words (Bateman 2008: 7), to acquire a “multimodal literacy,” that considers carefully “the multimodal document as a carrier of meaning that draws on visual, spatial and verbal presentational modes in combination and co-operation.”

A further step should be to become sensitive and reflective about how multimodality is persuasive (either effectively or not) by responding to a context. Design and interpretation are always design and interpretation of situated discourses (see Bitzer 1968). For instance, by commissioning an ad designed to unmistakably associate the brand to Italian authenticity and freshness of the products, Panzani creates a world of the brand that is apt for one specific audience at one specific point in time leading the audience to purchase the products. A well-thought marketing strategy like this one results from a good knowledge of the context and the audience and could be unsuccessful with other conditions. Thus, “any attempts to come up with simple, universal recipes for successful and effective presentations are doomed to failure – there are no simple, universal explanations for responses to or the effects of persuasive communication” (Jackob et al. 2011: 265). We embrace the solution Kjeldsen suggests, which is to go from multimodal literacy to multimodal rhetoracy:

The solution is for us to stop thinking in terms of technology and begin to think rhetorically. As speakers, we should not be thinking of how to fill in a template. We should be thinking of purpose, goals and means. What do we want to achieve? How can we best achieve it? What tools will help us best? And what are the technological limitations and possibilities? … As in any other form of communication, the speaker, before speaking, should emulate Cicero. He/she should perform a rhetorical analysis of the situation, the audience and communicative tools in relation to intention … The time has now come for us also to think about Media Rhetoracy, in other words rhetorical competence. (Kjeldsen 2006: 12)

2.2 The visual and the multimodal in argumentation studies

Simultaneously to the above-presented studies within the field of multimodal discourse analysis, scholars from argumentation studies started to ponder on the existence of visual arguments. The 1996 special issue of the journal Argumentation & Advocacy on visual argument marked the start of this interest, but not without criticism and resistance (Fleming 1996; Blair 1996; Johnson 2003). In their pioneering essay, Birdsell and Groarke (1996: 1) argue against the skeptical position that images cannot be arguments because they are indeterminate, vague, arbitrary, and intrinsically ambiguous (e.g., Blair 1996). Yet, they say, words as well may be vague and ambiguous in natural arguments and one cannot deny that there might be some debate about the interpretation of verbal arguments as well. Other features considered to be obstacles to the existence of visual argument were the nonlinear arrangement of elements in images, their – apparent – lack of propositionality, and their implicitness. The first publications on visual argument focused on either arguing for or arguing against these positions with little room for new theoretical proposals and analytical frameworks.

The transition from visual argument to multimodal arguments was not without obstacles either. As images often work in combination with verbal texts to advance premises and conclusions, scholars arguing in favor of visual argument often employed examples where an interplay between semiotic modes conveys the message. Multimodal documents were considered as a counterexample of visual argumentation by Fleming (1996) and Blair (1996). Lake and Pickering (1998) soon pointed out that the combination of words and moving images is a “strength” and it “does not mean that only discourse carries an argument and that images at best reinforce or serve as evidence for discursive claims. On the contrary … there are many possibilities.” Some types of relations that the verbal and the visual can establish in argumentative discourses are identified in Roque (2012): the visual and the verbal may present parallel and redundant argument, the visual and the verbal construct jointly the argument, the argument is constructed through an opposition of the verbal and the visual. However, these categories remain at a descriptive level and do not consider the contextual constraints of persuasive communication.

As it is the case for the field of multimodal discourse analysis, a rhetorical approach represents a step further in the studies of visual and multimodal argumentation. Some scholars claim that images and multimodal combinations can be fruitfully conceived as enthymemes. In the rhetorical tradition, an enthymeme is a syllogism where some premises are tacit: “[the enthymeme] is deduced from few premises, often fewer than the regular syllogism; for if any one of these is well known, there is no need to mention it, for the hearer can add it himself” (Aristotle, Rhetoric, I, 1357a)

According to Bitzer (1959: 407), “to say that the enthymeme is an ‘incomplete syllogism’ – that is, a syllogism having one or more suppressed premises – means that the speaker does not lay down his premises but lets his audience supply them out of its stock of opinion and knowledge.” The suppressed premise(s) can remain unstated because they are shared opinions, values and pieces of knowledge that already belong to the common ground (Rocci 2006).

For Smith (2007), images can be fruitfully seen as enthymemes precisely because they depend on the ability of the rhetor to find common opinions that the audience agrees with and that the audience is required to actively retrieve. Because of the pictures’ “potential for semantic condensation” (Kjeldsen 2012: 241), “they offer a rhetorical enthymematic process in which something is condensed or omitted, and, as a consequence, it is up to the spectator to provide the unspoken premises” (Kjeldsen 2012: 241). In multimodal discourse formats, images together with other semiotic resources provide cues and directions for a correct interpretation of enthymematic arguments. This is the case, for instance, with the cognitive schemes provided by metaphor and other tropical means (Kjeldsen 2012; Pollaroli and Rocci 2015).

These are the starting points to argue in favor of a sensible marriage between multimodal discourse analysis and argumentation theory with a division of complementary tasks. Multimodal discourse studies focusing on the semiotic modes and their interaction in the orchestration of discourses and argumentation theory focusing on the link between premises and conclusions. Rhetoric represents here the bridge between the two by placing multimodal discourse items in contexts and showing their persuasive functioning and force.

3.1 The thinness of multimodal argument: The problem of under-specification

Paul van den Hoven and Assimakis Tseronis begin this special issue with discussions in favor of considering multimodal argumentation as proper argumentation.

In the chapter “Discussing discourse modalities in argument theory,” Paul van den Hoven exploits Anthony Blair’s opinion that purely pictorial communication is more indeterminate than purely verbal communication to start a discussion about the implication of “a hidden paradigm” of argumentation theory, which corresponds to the idea that arguments “can be reconstructed in a fairly straightforward way” from the verbal material of discourse. This discussion is not just a way to show that multimodal argumentation exists and needs room in studies on argumentation (something that has already been proven in other publications). As van den Hoven points out, “uncovering the somewhat hidden paradigm that Blair may have in mind as a reference point … helps us gain insight into the implications involved in fully acknowledging multimodality as a means to convey argumentation for the development of argument theory.”

The author agrees with those who doubt the possibility for visuals and multimodal artefacts to be argumentative about the fact that the degree of indeterminateness in multimodal discourse is higher than in prototypical discourse composed of ordered sets of propositions. But he doubts the existence of such prototypical discourse types. Indeterminateness is a feature of multimodal discourses and it results in, first, the audience being asked to be more active in determining the meaning of a multimodal discourse and to find what is relevant by themselves. Second, in multimodal discourse many elements are implicit and mixed in complex discourse worlds. Third, creative schemes are employed in order to construct a discourse, such as metaphors and narratives.

Van den Hoven concludes that having argumentation scholars maintaining the hidden paradigm would result in implications for argumentation theory as an academic discipline because it excludes the possibility for some multimodal discourse formats to be considered proper argumentative items.

In a similar vein, Assimakis Tseronis suggests a way to abandon the strict concepts coming from a narrow idea of argumentation in order to better welcome multimodal argumentation. In the chapter “Multimodal argumentation,” Tseronis employs an ad campaign for the promotion of the newspaper The Guardian to argue that “argument is neither verbal nor visual since argument is not to be defined on the basis of the verbal, visual or other semiotic means by which it is realized in communication.” After a close reading of the literature on visual argumentation – pointing out its achievements and its limitations – Tseronis proposes overcoming the distinction between the verbal and the visual in the study of argumentation and to adopt, instead, “a multimodal perspective,” which should take into consideration also the research developed in the field of multimodal discourse analysis. This means, in the author’s words, that “the argument is to be reconstructed by taking into account the interaction of the verbal and the visual mode and by paying attention to choices made regarding the visual form composition, not by merely reading off the standpoint from the verbal mode and the arguments in support of it from the visual mode.” For Tseronis, acknowledging the “multimodal nature of communication” would be a step forward towards opening the research to more interesting questions that simply asking whether and how images are arguments. More interesting questions may include: “what is the argumentative relevance of the various modes that may combine in genres of different argumentative activity types and how can we go about analyzing and evaluating argumentative discourse, in which more than one modes combine?”

In order to show that “the argumentative analysis of multimodal discourse is … not merely seeking to identify whether a picture or some visual element plays the role of the standpoint or the role of the argument but how choices made from the affordances of the various modes can be said to contribute to the on-going argumentative procedure,” Tseronis analyzes some examples from an advertising campaign for The Guardian. The author shows that the layout mode of the print advertisements is fundamental to read the main argument provided in support to the standpoint “Buy the Guardian” and that the double illustrations by Noma Bar are not merely depictions of the referent of the propositions used in the arguments, but they help navigate the message.

3.2 The thickness of multimodal argument: The virtues of condensation

From Jens Elmelund Kjeldsen’s article “Visual Rhetorical Argumentation” we learn that pictures should be studied by taking into account their specificity. Kjeldsen argues for the existence of visual argumentation and supports two related claims: (1) the study of visual argumentation is possible by considering that argumentation is a cognitive phenomenon embedded in a specific situation and by introducing the notion of symbolic condensation; (2) not only must the study of visual argumentation consider pictures to be as relevant as language is but also it must take into account that the aesthetics of pictures is different from that of words. Indeed, “[b]ecause of the phenomenological quality of pictures, their visual plenitude and semantic ‘thickness’ disappears if we reduce the pictorial representation to only theoretical concepts, cognitive phenomena or simple verbal propositions.”

Indeed, pictures are characterized by “symbolic condensation” because they condense “many different ideas into one, so that the effect and meaning of a picture is grasped in one single instant – in a blink of an eye, so to speak.” This results in the strength of pictures, which comes from their immediacy of understanding, potential instantaneous reception, possibility to cue and evoke thoughts and feelings, semiotic richness, and simultaneous coding. According to the author, symbolic condensation can be both emotional and rational. It is emotional because a picture of an event tends to activate the same emotions of a similar event in real life; the emotional trait can be affected (and might be manipulated) by, for example, the framing and the angle. Symbolic condensation is also rational because it corresponds to an enthymeme, which must be processed and filled in by an active audience within a situated interaction. Indeed, Kjeldsen supports an active participation of the audience in the reconstruction of enthymematic arguments. And scholars should “examine the interplay between text, context, and reception” by supporting their studies with reception analysis.

Kjeldsen concludes by pointing out two faulty approaches to the study of visual rhetoric and argumentation that should be avoided. First, we should avoid the reduction of the study of the rhetoric of images to a tropological or ornatus approach (cf. Genette 1970). Kjeldsen does not claim that the study of visual tropes and figures should not be carried out (see Kjeldsen 2012) but he is aware that other aspects of rhetoric should be taken into consideration, for instance the rhetorical situation, visual aesthetics, values, and topoi. Second, we should avoid the interpretation of images as texts to be read and decoded as if they were verbal texts. Pictures lack syntax and the meaning-making units are difficult to distinguish. The coding of picture is weak, they do not have double articulation, and they tend to be ambiguous.

3.3 Multimodal meaning-construction and the logical structure of arguments

From the very beginning of her contribution, Janina Wildfeuer points out a very important step made in the studies of multimodal discourse analysis, especially those interested in audiovisuals: “film analysis, and multimodal analysis in general, is no longer seen as simply decoding the various semiotic resources involved in the meaning construction, but as asking for inferential processes of reasoning about the best and most plausible interpretation.” Indeed, the structure of films provides constraints on their interpretation, which “operates according to so-called textual cues … in the artifact that clearly guide the recipient’s imagination and hypothesis-making.” This is an abductive and defeasible inferential process.

The author applies the “logic of film discourse interpretation” based on discourse semantics in order to “outline the discursive and rhetorical structure of filmic text and to retrace the inference process of the recipient.” In particular, this enables one to:

1. analyze the semantic content of films by illustrating in a box each event of the audiovisual discourse according to the recipient’s knowledge of the world and the interplay of semiotic resources (the box shows a list of the discourse referents represented both visually and aurally tagged with a variable to make it possible to describe dependencies; the eventuality is described as a propositional verbalization although it remains a defeasible one rather than a strong one; a logical operator signals that the inference of the eventuality is a defeasible relation);

2. describe discourse relations between the narrative events that maintain the film’s coherence. Among the set of discourse relations provided by Asher and Lascarides (2003), Wildfeuer (2014) identifies the following in filmic discourse: Narration, Elaboration, Explanation, Result, Background, Parallel, Contrast;

3. understand how the recipient works through the text and makes inferences, which are constrained by (1) and (2).

The logic of film discourse interpretation is a tool particularly suited for the analysis of dynamically unfolding discourses where recipients have to make a leap “from the last event to an event somewhere in the discourse before,” such as El vendedor de humo, which Wildfeuer chooses to analyze. Here recipients have to bring together the shots – thus we see the importance of montage in films – and their narrative events and infer a coherent relationship between them. Then the recipient is able to evaluate critically what s/he has seen before.

The inferential structure of metonymical multimodal arguments in campaigns is the focus of the article “The argumentative and rhetorical function of multimodal metonymy” by Andrea Rocci, Sabrina Mazzali-Lurati, and Chiara Pollaroli. By analyzing two social campaigns employing various channels to communicate their messages, the authors show that metonymy shrinks a complex causal chain and highlights the most relevant parts of it. A combination of Blending Theory by Fauconnier and Turner (1998, 2002) and of the Argumentum Model of Topics by the Lugano school of argumentation (Rigotti and Greco Morasso 2010; Rocci 2017) is employed to reconstruct the elements of the framed situation triggered by a multimodal metonymical advertising message and the inferential path that links them to a thesis (which is often implicit and must be reconstructed from the available multimodal elements). Metonymy is shown to be isomorphic to argument schemes of the causal type and is shown to work rhetorically by choosing to highlight those elements of a causal chain that are close to the heart of the audience.

3.4 Multimodality in the analysis, design, and support of argumentative activities

The article by Marta Zampa and Marina Bletsas describes an argumentative activity about a multimodal object in a newspaper. They “present the argumentative analysis of a deliberative discussion at the main Italian-language newspaper in Switzerland, Corriere del Ticino, concerning the design of a multimodal news item labeled ‘picture news’.” Picture news are multimodal news items made up of a photograph, a capitalized caption above it together with a short article below it. They have a central position in the front page of a newspaper and represent a strategic choice for a newspaper in terms of salience of information and appeal. Zampa and Bletsas show how the news values of Corriere del Ticino and the dynamics of the interaction in the newsroom come into play when a decision about which picture news to choose is to be made. Among the criteria of the newsworthiness of pictures that “guide decision-making concerning the creation of picture news during editorial conferences,” the “availability of pictures of an event” prevails over other criteria (e.g., authoritative source of the picture and technical quality of it, aesthetic appeal of a picture, no pictures that might hurt the audience’s feelings) in the case study offered as an example in the chapter: “the image that is eventually selected is not aesthetically appealing, but at least depicts an event which has some impact on the Ticinese population and of which there are original pictures at disposal.”

The article by Silvia De Ascaniis, Sara Vannini, and Lorenzo Cantoni titled “Argumentation in participant-driven photo interviews” deals with pictures as vehicles to elicit an argumentative activity and obtain multimodal argumentation with the same pictures taking a significant role in the design of it. With the aim of understanding what people working in Monzambican Community Multimedial Centers think about the social value of these centers, participants were asked to take pictures of something they liked, something they disliked or wished to improve, and something they perceived meaningful and representative of their center. The pictures were then used “as catalysts of the argumentative discourse, as anchors to start a reflection on the meaning of the CMC for individuals and for the community.” As part of a broader research project on the social significance that different agents attribute to technology in developing countries and the possibility of achieving sustainability in the centers, in De Ascaniis et al.’s article, the focus is on the function pictures have in the construction of argumentation. By analytically reconstructing the interviews, the authors are able to check the argument role that photographs play and their semiotic function in relation to the verbal text. The findings showed that “photos played a substantial role in the argumentation about CMC, in that they were not decorations of the discourse; rather, they directly contributed to the propositions working as premises and standpoint. Photographs accomplished one of three roles: (a) they metaphorically represented the standpoint; (b) they gave evidence for the argument; (c) they metonymically represented one aspect of the argument. In the second and third case, photos functioned as factual premises in the argumentation.”

Sabrina Bresciani and Martin J. Eppler present a study on the use of diagram representations of argument to help to argue together. The tools for argument diagramming that the two scholars examine present different forms of visualization and were designed for different purposes: Araucaria is a formal and rigid-structured tool designed to facilitate the teaching of philosophy and critical thinking and to better evaluate arguments; Compendium allows for the pictorial representation of issues, claims, and arguments in a flexible format and with the employment of colors and icons; Debategraph is an online tool to which different users can contribute and it allows for the creation of cross-links between maps. The two contributors wish to describe the “visual aspects of argument maps in order to provide a vocabulary for discussing, selecting and designing the appropriate argument mapping technique for specific purposes” and find out “which are the most relevant visual aspects to consider when designing, evaluating or improving argument maps.” In order to do so, Bresciani and Eppler propose a socio-visual framework “which can be used to better design, understand, evaluate, and use argument maps in collaborative settings for decision making purposes.” The factors composing their framework are (1) visual impact, (2) clarity, (3) perceived finishedness, (4) directed focus, (5) facilitated insight, (6) modifiability, (7) discourse management. The authors conclude that argument maps can “better serve as collaborative argumentation catalysts” if designers take into account the seven factors of the framework in accordance with the context and purpose of use of the tool and if users of diagramming tools employ the seven dimensions to question the usability and effectiveness of the tool for collaborative argument mapping. Of course, collaboration between designers and users can result in an improvement of these tools.