- 1 For instance, Orquin and Holmqvist stress that “It can be very tempting to think that eyetrackers r (...)
1In a recent eye-movement study of comics reading patterns (Mikkonen and Lautenbacher 2019), we studied possible explanations for readers’ scanpaths that deviated from the default Z-path reading order at the beginning and end of the reading task. Thus, we identified more “global” saccades vs. “local” saccades and argued that this reading phenomenon could be an indication of global attention that explores the informative and perceptually salient areas of the stimuli. We believe that these findings can serve as a corrective to theoretical assumptions about reading comics. Nonetheless, at the same time, we continue to feel somewhat suspicious of some of the general claims made about higher-level cognitive processes in current eye-tracking research. The main reason for our skepticism is what in cognitive psychology is known as the eye-mind assumption. This assumption states that the eye remains fixated upon some object, such as a word, until processing has been completed and, consequently, that fixations are coupled with the process of attention. Despite being a premise for much research with this technique, the assumption is also frequently problematized, and in some cases falsified.1
- 2 The quantitative data is based on a “Transition Matrix” that Olli Philippe Lautenbacher created for (...)
2In interpreting eye-tracking research data, the eye-mind assumption is also reflected in the challenges of identifying gaze behavior that may be significant in terms of narrative comprehension. To illustrate this dilemma, let us take the example of Areas of Interest (AOI) that evoke more deviations than the surrounding AOIs in the same stimulus.2
3Figure 1 presents a double-page example with a rather conventional layout style, from Pascal Rabaté’s Un ver dans le fruit. We identified obvious differences between the AOIs regarding the number and range of eye-movement deviations. For instance, while in three of the AOIs (1, 4, 7), out of the total of 11, the most common track followed by at least three fourths of the readers (74–90%), was the Z-path, in seven other AOIs, only 35–50% stayed on it, although it was, still, the most frequently chosen path. We may presume that these differences are relevant in terms of narrative processing. Yet, how to substantiate that claim?
Figure 1: Priest in his cathedral. AOI-manipulated spread from Pascal Rabaté’s Un ver dans le fruit, 1997, p. 36–37.
© Pascal Rabaté, Vents d’Ouest et Glénat, Paris.
- 3 Furthermore, AOI 5 invited the largest range of deviations among all the panels in this stimulus (9 (...)
4The problem is not that we cannot find relevant explanations, but that these are of different kinds, and their validity needs to be assessed somehow. To use AOI 3 as an example, where the Z-path “default” movement was particularly weak (35%), we may list at least the following potential explanations for the deviations: 1) Narrative (explanation): the reader’s need to comprehend the beginning of the sequence through overview-preview movements may explain the regressions and saccades across AOIs 1–3; 2) Narrative: AOI 3 and 1 share a subjective over-the-shoulder perspective of the same character, thus inviting saccades across the upper row; 3) Conventional reading order: the movement from AOI 3 to AOI 4 conforms to the Z-path reading pattern; 4) Compositional and narrative: the panel’s place at the edge of the page, and the repetition of the same figure, may explain some readers’ move to AOI 6 and 9 on the next page; and 5) Compositional and narrative: the movement to AOI 5 may be triggered both by the combined effect of what Neil Cohn calls horizontal stagger and blockage (2013b), and the panel content that situates the action in a larger setting, i.e. the altar in the nave of a cathedral.3
5This abundance of explanations need not be a problem: we might want to argue that all of them together contribute to the range of deviations. The variety of possible explanations also reflect the reality that there are a lot of different simultaneous pressures acting on the reader. The difficulty in distinguishing between narrative and layout explanations, further, suggests that the interaction between image content, words, and layout is a key question in reading comics. Yet, should one perhaps hesitate a bit longer before making the interpretive move from gaze behavior to narrative comprehension? To consider deviations in one AOI at a time can be problematic since options at any one narrative moment may be affected by what has been processed before. In eye-tracking data, the relevance of larger units of semantic and cognitive processing, such as image sequences and groups that depict narrative scenes, may be detected in regression and see-saw patterns, refixations, and more global saccades, but their relevance in terms of narrative comprehension requires interpretation. Also, if saccades correspond with shifting attention, it still remains open how we get from implications of attention to narrative processing that occurs at the panel level, i.e. through the relation of panels. In addition, the equation of AOIs with panels, which was the preprocessing move in our study, may neglect distinct areas of interest within one AOI. This may be amended, perhaps, by a heat map presentation (see Figure 2) that displays the distribution of looking over the stimulus regardless of panel borders, and dwell time analysis, but that data again must be interpreted in terms of narrative salience and meaning.
Figure 2: Priest in his cathedral. Heat map representation.
© Pascal Rabaté, Vents d’Ouest et Glénat, Paris.
6Furthermore, our interpretation of instances that evoke more deviations may be affected by several other factors, such as the participants’ fluency in reading comics, or the reader’s preconceived notion of narrative salience. By the latter we refer to those elements in the images or in the composition that seem particularly significant for understanding the narrative event. In this respect, our empirical experiment confirmed the not so new insight that readers fixate especially on words and characters (or other foregrounded figures).
7Despite all the valuable evidence and data that eye-tracking and eye-movement reconstructions can provide about points of attention, saccades, and reading order, they cannot easily answer two crucial questions: 1) How do more top-down cognitive processes, such as reading for the narrative, influence how and what readers see? And 2) How does the processing of narrative content affect eye-movements and gaze behavior? Such reconstructions can, at best, reveal possible indications of narrative comprehension, but that again requires some preconceived notion of what makes a narrative, or what may be recognized as a narrative.
8One highly conventional way to recognize a text as narrative, and start to read it accordingly, are anthropomorphic figures in some situation or an event. As various theories of narrative hold since Aristotle, the process of following the characters’ actions is constitutive of narrative (or tragedy) (see also Altman 2008: 15–17). However, while eyetracking tests make clearly evident how characters are the readers’ main areas of attention and interest, the many functions that characters can serve at the same time—mimetic, thematic, plot-related, metafictional, and so on—are difficult to capture by way of eye-movement reconstructions. This multidimensional, or multichromatic, quality of characters should be given more attention in future empirical research.
9In the following, we leave the empirical work aside for a moment and explore the coreferential function of characters or other repeated figures in comics. In linguistics, two expressions, or constituents in a sentence, are coreferential if they refer to the same entity (Crystal 2008: 116–117). Thus, for instance, the names Captain Hawdon and Nemo refer to the same individual in Dickens’s The Bleak House, as do also the personal pronouns he and him, or descriptions such as “Krook’s dead lodger”. Sustained coreferentiality throughout a lengthy text helps the reader to read for the narrative, e.g. understand the narrative event and situation and, thereby, comprehend the story as a coherent and meaningful whole–even if many names for the same character in Dickens may also involve mysteries to be solved.
- 4 In her review of Mikkonen’s The Narratology of Comic Art, Braithwaite (2018) took the liberty of co (...)
- 5 “On définera comme solidaires les images qui, participant d’une suite, présentent la double caracté (...)
10The idea that recurring characters can, as they suggest a visual bridge between the panels, simultaneously serve characterization, i.e. cues for a mental model of a character, and reading for the narrative, is based on our earlier insights in comics narratology (Mikkonen 2017: 90–108; 183–185). Here we develop these insights further, and have decided to call the character’s, or other foregrounded figure’s, narrative-constructing repetition figural solidarity (FS) as a critical adaptation of Thierry Groensteen’s concept of iconic solidarity (“solidarité iconique”, from here on IS).4 Groensteen famously defines IS as “interdependent images that, participating in a series, present the double characteristic of being separated […] and which are plastically and semantically over-determined by the fact of their coexistence in praesentia” (2007: 18; 1999: 21).5
- 6 To mention just some of these uses: Meskin (2017); Postema (2013: 46–48; 55; 72–73); Molotiu (2013)
- 7 These are spatial complementarity, perpetuation, rhythm, and configuration. Instead of suggesting a (...)
11With the IS concept, Groensteen named an essential principle of sense-making and comprehension in comics, and derived from this principle the useful, and much used6, distinction between two basic kinds of panel relation, restricted and general arthrology (the latter inviting the process of braiding). We contend, however, that Groensteen did not fully explore the scope of the concept, that the definition of IS remains vague despite its intuitive appeal and applicability, and further that the notion subordinates the visual content in the images to the spatial arrangement of panels. In addition, the specification of this notion in Bande dessinée et narration (2011) by way of four modalities7 only manages to make the concept vaguer and more abstract.
12Our main critical issue with the IS concept has to do with the limited relevance that this concept grants to the intra-panel referential content of the images placed in a series or sequence, while it privileges the processing of spatial relations, and the interdependence of panels, as the fundamental condition of meaning (Groensteen 1999: 25–26). We contend, more precisely, that due to the emphasis on the spatial structure of the panel organization in the “spatio-topical apparatus” of comics, the IS notion disregards the way in which the visual content of the images can be more than “just” iconic or plastic content. In other words, what we want to highlight is how repeated visual content, and especially figures (to be defined in the next section), can make possible the very operations of this principle, at least in narrative comics.
13Groensteen’s inconsistent use of the term “iconic” also needs to be noted. On the one hand, the iconic in IS refers to the panel as a unit of composition within the spatial organization of the multiframe that makes the work (2007: 20; 23–24). The iconic in this sense does not have to have any content but is a unit of layout and spatial organization that is a precondition for meaning in comics. On the other hand, the “iconic” is also an element of panel content, along with plastic elements (1999: 5) and verbal signs. The iconic, then, also means for Groensteen the recognition of a visual similarity, continuity, alteration, or other meaningful visual relation between panels.
- 8 In terms of Roman Jakobson’s model of language functions, the plastic focuses on the poetic functio (...)
14In this latter sense, the definition of the iconic follows the Belgian collective of Groupe μ and their systematic semiotic rhetoric of the image that distinguishes the iconic and the plastic as two different aspects of the visual sign and as distinct ways of attending to visual signs. Thus, for Groupe μ, and for Groensteen who draws from their work in Système (1999: 4–7; 55; 64), the plastic refers to the visual sign as a plastic form (e.g., a circle as a visual shape), while the iconic concerns the image as a visual representation (e.g. a representation of a circle) (Groupe μ 1992: 115–116; 120–122). As ways of attending the image, furthermore, the focus on the plastic qualities of visual signs involves looking at the forms, lines, spaces, texture, and colors of images, while the focus on the iconic is to perceive images as representations of diegetic objects such as characters. These two modes of attention are not in opposition to one another, as Groupe μ further argues, but can be perceived as “solidary” to one another (1992: 118–119).8
- 9 The process of attributing traits to characters, construing identity/unicity for them, and trying (...)
- 10 We therefore adhere to a multichromatic view of character as consisting of various dimensions in i (...)
15By contrast, our FS concept focuses on the iconic in the sense of visual representation, but with a specific emphasis on how iconic visual content not only represents something but can help to move the story forward. Similarly to Groupe μ, we also see that FS involves a way of attending to visual signs. As such, our reframing of IS rests on important assumptions about reading comics and following characters. First, we hypothesize that FS involves a fundamental higher-level cognitive mechanism, in which characters, or other reoccurring figures in the image sequence, provide the reader with access to narrative and a means of continuity construction. Secondly, we presume that while FS may in actual reading experience be inseparable from the process of characterization, i.e. reading for character9, it can be distinguished as a separate top-down cognitive process for theoretical reasons. In this process, we further presume that readers follow recurring characters not only for their own sake, but also in order to make sense of the narrative, i.e. they read for the narrative.10 As such, FS refers to the process in which readers create a chain of reference as they process recurring visual elements that allow them to link up the images and text, and thus build a meaningful narrative context.
- 11 In Système, Groensteen also mentions Krafft in passing among the scholars who have emphasized the m (...)
16One important predecessor to our concept is Ulrich Krafft’s semiotic theory of reference (1978) that Groensteen dismisses for its focus on “sub-entities” of iconic signifiers (1999: 4).11 Krafft formulates the notion of a chain of reference based on the repetition of visual elements in the panels, and perceives the function of these signs in terms of co-reference. He thus defines the setting function (Setzung) of signs, for example the first appearance of a character, typically in the foreground of a panel, which then establishes a chain of reference for subsequent panels (1978: 18; 21; 28). Krafft, furthermore, allows for gaps in sequence and resumptions of interrupted chains, thus anticipating Groensteen’s distinction between restricted (linear) arthrology and nonlinear braiding. Later Mario Saraceni, in his linguistic analysis of image repetition, even if this approach is limited to the question of panel-to-panel transition à la Scott McCloud, usefully distinguished between the textual and the cognitive component of relatedness. These components, Saraceni further argued, can interact by way of repetition, collocation and inference (2001; 2016).
- 12 Gavaler and Beavers replace McCloud’s term “closure” with their term “hinge” in their later textboo (...)
- 13 Including, for instance, Bornens’s (1990) study on how children, from the age of four years on, lea (...)
17More recently, both theoretical and empirical approaches have employed concepts closely related to our notion of FS. Among these we can list Tseng and Bateman’s (2018) empirically grounded investigation of cohesion chains through “reoccurrence” relations, Bateman and Wildfeuer’s (2014: 385; 389) formulation of the discourse coherence relation that necessitates a shared topic between the panels, and Gavaler and Beavers’s term “recurrent closure,” which describes the way in which two images reference the same subject (2020: 187–188)12. Similarly, Neil Cohn’s notion of “continuity constraint” (2021), which refers in his Visual Language Theory (VLT) to the way in which a “comprehender” recognizes that “the characters and objects in one image are the same referential entities repeated in subsequent images” (2021: 16), and which draws from Saraceni’s and others’ earlier ideas,13 emphasizes the importance of continuity-construction in the interpretation of visual sequences. We invite the numerous comics researchers interested in coreference to rally now under a single term, and we modestly propose figural solidarity for that purpose, both for the sake of continuity with seminal past work and because we believe that figures, in particular, are the most viable direction for future investigation into the formal semantics of comics images.
18Before proceeding, we will pause to consider a formal definition of figure, and at the same time to add some intuitive support for the concepts of figure and figural solidarity. In Chris Ware’s charming comic “A Short History of Cartooning,” the three-panel sequence titled “Our New AltarpieceTM” shows an artist presenting his painted sequence to two red-robed and mitred figures—Catholic cardinals, presumably. “And over here are the twelve stations of the cross,” says the artist, beaming proudly. The cardinals are bemused. “Yes, but,” says one, “why do you paint twelve Christs, when we know there is only one?”
19The fictional cardinal’s misreading of the image sequence highlights something so fundamental about how we read comics (and similar media) that it is effectively invisible. The cartoonist may multiply images without thereby multiplying the conceptual entities evoked in our mind. That is, images may co-refer. In each of 12 scenes of the crucifixion narrative, separate painted figures portray the same person. As we readers parse a comics sequence, we interpret the figures so fluently that it can be difficult even to notice that anything about this situation merits explanation. It’s as invisible and necessary as air. But there is something to explain: a detailed theory of reference has so far eluded comics scholars.
20Quite elaborate theories of reference already exist for linguistic expressions; current comics researchers differ considerably in their attitudes toward linguistic theory as a potential guide. The most striking and extreme position is Neil Cohn’s VLT, which claims that comics are necessarily drawn in specific “visual languages” just as novels are written in languages such as English or French, and that “visual language” is not metaphorical, but exactly parallel to linguistic-type languages, down to the presence of morphological and syntactic structures (2013a; 2021). Saraceni uses the word language but makes it plain that what he means by the term is “essentially equivalent to the semiotic concept of code, or system of signs” (2016: 116). Bateman and Wildfeuer, in direct opposition to Cohn, argue that “[v]ery few of the properties exhibited by grammars of natural language in fact carry over to visual media such as comics” (2014: 374). To us it seems clear that narrative comics texts and purely prose texts both present readers with interpretable saussurean signs, and that images may direct readers to certain mental concepts just as reliably as words do. When we hear or read linguistic expressions like “Jesus” and “stations of the cross,” the words evoke certain culturally shared concepts. And images like Figure 3 below evoke the same concepts.
Figure 3: Crucifixion scene.
- 14 Thanks to an anonymous reviewer for emphasizing this point.
21Incidentally, we hope no reader will suppose us to have either blasphemous or devotional intentions! Having begun with Ware’s altarpiece, it’s convenient to continue with a narrative likely to be familiar. Consider, then, a full rendition of the crucifixion story in words only. It would include numerous repetitions of proper nouns such as “Jesus” or “Christ,” appropriate pronouns, and other descriptive or conventional noun phrases such as “the condemned man,” “Our Lord,” and “the broken body,” all of which would be understood by the reader as referring to the same individual. The same (or a very similar) narrative rendered in sequential images might likely include panels like those in Figure 4. Specific drawn figures within these panels are signifiers pointing us to the same signified character as the various noun phrases did in the purely linguistic narrative. That is not to say that the informational-expressive apparatus of prose- and image-based texts are parallel in every way.14 For example, a narrative scene evoked over a series of comics panels naturally includes an abundance of physico-spatial detail: information is repeatedly provided about, e.g., a character’s hairstyle, clothing, current posture, etc, even where this is not particularly relevant. A prose author might provide such information only once or not at all. The comics medium has its specific affordances, of course. Our point is merely that the referential functions of certain components are comparable.
Figure 4: Three stations of the cross.
- 15 See also Gavaler and Beavers’s notion of “diegetic recurrence” (2020: 188) and Stainbrook’s “concep (...)
22This functional parallel between noun phrases and repeated figures in panels has been noted before, by Mikkonen (2017: 183). The coreferentiality of numerous sequential figures provides both cohesiveness and driving force for visual narratives. The storyworld is populated with concrete characters and objects; any of them may be introduced to the reader in the form of a drawn figure, and what befalls these entities at later points on the narrative timeline may be depicted by subsequent coreferential figures. Note that the actual marks on the page might be far from identical in appearance; their identity is conceptual, and diegetic.15
23We believe that the concept of figural solidarity can usefully replace iconic solidarity for many purposes. We have already discussed Groensteen’s individual use of the term. By now, though, the reach of this very popular phrase goes far beyond Groensteen’s own use. Over the past 20 years this term/concept has been in wide use by a great variety of people, some of whom have used it far more loosely than Groensteen, as a simple nod in the direction of… whatever is the mysterious something in the image track of comics sequences that gives them coherence and artistic unity. It is our hope that the idea of FS may eventually give considerable explanatory traction to Groensteen’s initial impulse, and that it inaugurates a broad research program into the visual semantics of comics, both for careful formalists like Groensteen himself, as well as others who may adopt the idea more informally.
24We chose the word “figure” deliberately, with two relevant senses in mind. It can mean body, especially (but not only) the human body, as in figure drawing and painting. It can also mean any salient representational pattern which we pick out among background material, as in the figure-ground distinction popularized by the Gestalt school of psychology. Figure 5 presents a well-known optical illusion in which you can either see the single image of a lamp or two faces in profile, depending on whether you focus on the white area or black areas as foreground.
Figure 5: Figure-ground optical illusion.
25Next consider the simple line drawing in Figure 6, another well-known optical illusion. The mark on the page is the signifier, but it can map to either of two conceptual interpretations: the rabbit reading or the duck reading. (This is very much analogous to the linguistic ambiguity of an English word like “bank,” which can signify either a financial institution or a location at the side of a river.)
Figure 6: Duck-bunny optical illusion.
26We look at marks on the page and we are predisposed to see entities in them, recognizable objects. Including that special subclass of objects that we recognize as animate beings like ourselves! Donald Duck and Bugs Bunny (for instance) are suitable figures onto which we can imaginatively project ourselves, because we posit them as having sentience and volition like us.
27Thus, as a first pass at a formal definition, let us try Definition 1.
28Definition 1: a figure is any discernible diegetic entity that the comics reader can pick out from among all the marks inside the comics panels.
29It is clear that, typically, the entities in which we are most interested are animate creatures, especially humans, but animacy may not be an absolute requirement. Typically, a comics panel is going to have one or more figures in it. Typically also, the panel will contain other material besides the figure(s), so that the reader must pick them out from the background. A minimal drawing such as Figure 6 can certainly constitute (the interior of) a comics panel all by itself, but minimalism to that degree is practiced only by some cartoonists some of the time. Generally, panels are busier than this. There’s no obvious upper limit to how many figures a single panel could contain, given a sufficiently fine resolution and large enough page.
- 16 A. Molotiu’s Abstract Comics may be the most famous reference. See also many examples discussed in (...)
30Clearly it is possible for a panel to contain zero figures: many artists sometimes interject solid white or solid black panels as indicators of time passing with no action and no new sensory input. Similarly, whole panels may at times be devoted to abstract content that functions in some expressionist or symbolic way to evoke mood or meaning but without direct graphic reference to any entities of the storyworld (see also Mikkonen 2017: 90). Pushing further in this direction, let’s imagine the creation of a complete comics story in which no panel contains any figure (all storyworld entities would be introduced only via the linguistic track). This would undoubtedly create a mannered or avant-garde effect, but clearly it could be done. In such a case, and perhaps also in the case of certain abstract non-narrative comics,16 FS arguably does not come into play. We will leave it to others to discuss such edge cases; we are confident that FS’s area of coverage is broad.
- 17 Jackendoff’s Conceptual Semantics is indeed an ideal place for comics scholars to seek insight, sin (...)
- 18 See Cohn (2013a: 28–30) for details and references.
31Language or Shmanguage? Figure 7 below, reproduced from Cohn (2021: 6), lays out the “grammatical architecture” Cohn proposes for VLT. The concept of a tripartite parallel architecture with meaning as one component and syntax (or combinatorics) as another derives from linguistics, and in particular the work of Ray Jackendoff.17 Cohn’s model, however, generalizes the linguists’ usual third component (phonology) to modality, which he says may be instantiated either by the sounds of spoken language, or the hand gestures of signed language, or the images of comics. We would dearly love to see a semantic theory of comics which could actually account for the connections implied by the bottom tier of Cohn’s diagram, where the smallest graphic “units” (bottom left corner) combine predictably to yield larger meaning-bearing structures. The question is, what exactly are these smallest units? In the past, Cohn and some prior semioticians have used the word grapheme, analogous to the linguists’ phoneme, for the putative minimal graphic unit.18 In his more recent writings, however, Cohn has moved away from this term.
Figure 7: Cohn diagram.
32Bateman and Wildfeuer specifically reject the traditional grail quest of formal semantics: “in contrast to grammar, the units do not build formally on each other compositionally” (2014: 377). Rather than beginning with a set of primitive atoms and building upward from the ground floor, they move immediately to discourse-level analysis, in which “units” may be of any functionally relevant size. We are not quite so pessimistic that no compositional semantic analysis can be devised for comics images, but Cohn has not (yet) done it, and nor has anyone else. That does not mean it’s impossible, though, or that the quest is of no interest.
33However attractive we find Cohn’s vision, though, a caution is in order. It cannot be the case that when Cohn identifies a visual language, such as American Visual Language, aka “Kirbyan,” that he literally means Kirbyan is a language in exactly the same way that French, Nahuatl, or American Sign Language are languages. The linguistic term “natural (human) language” (NL) covers these latter three examples and thousands more, but it can’t apply to a VL. If Kirbyan were truly an NL, it would follow that it could be someone’s native language, that there could be a linguistic community of people communicating with each other only in Kirbyan, and that Kirbyan-English and English-Kirbyan dictionaries could be created. Translation is possible between any two natural human languages, with only minor difficulties. Certain nuances may be elusive, but translation works very well, preserving meaning to a high degree of precision.
34Speakers of any NL have acquired a mental lexicon, in which the words of that language are saussurean signs which map to meanings from combinations of physical signifiers (a bundle of sounds spoken aloud, or a string of printed letters on the page, or a sequence of hand movements in signed languages). In translating between NLs, we do not always have a one-to-one matchup of words; moreover, because languages can differ considerably in how efficient their syllable structures are, and how highly inflectional their morphology is, the wordcount may differ between the source text and the output. However, it is never orders of magnitude different. Cohn contends that comics readers are working with an internalized VL lexicon too, where meanings are evoked by physical marks combined into meaningful morphological units. Perhaps this is correct, but a VL lexicon cannot possibly be organized on the same principles as an NL lexicon. There is no clear VL counterpart to words, or the smaller, sub-meaningful units that compose them (graphemes per Cohn [2013a], simply “units” per Cohn [2021]).
35As one illustration of the difference between VL and NL, consider the following translation exercise. In Gareth Hinds’s graphic adaptation of Macbeth, the most famous soliloquy appears on page 115, “Tomorrow, and tomorrow, and tomorrow/ Creeps in this petty pace from day to day.” Wisely, Hinds has rendered these lines as English text, inside a speech balloon. The task of translating the lines into text-free Kirbyan would be a daunting one. Abstract, time-related concepts do not lend themselves to the affordances of comics, which is why even a cartoonist working almost exclusively with images will tend to drop captions like “5,000 years later” onto otherwise wordless panels.
36The basic idea and mood of Macbeth’s speech could no doubt be gotten across in images alone, supplemented with explanatory icons such as arrows; we’d particularly like to see Kevin Huizenga try his hand at it! But it would require many pages and the precision would be lost. A Kirbyan-English translator then confronted with Huizenga’s pages could not hope to translate it back into anything like the original English. In fact, we are not dealing with translation at all, but transmedial adaptation. To call a VL a language is a tempting for many reasons, including its vast expressive scope, its referential capacity. But we cannot simply assume that features present in NL will also be present in VL; in particular, we don’t get the VL morphological structure for free.
- 19 The claim that images cannot be easily segmented into linguistic elements such as sentences, phonem (...)
37The signifier side, the marks side, of VL shmanguages are not segmented neatly into wordlike things we can list in an alphabetized lexicon. They are not easily segmented at all, and that is the main stumbling block we will face in creating any semantic theory of comics.19 There is no clear least meaningful unit, and there is no clear discrete set of sub-meaningful units. No one has yet proposed a convincing graphemic inventory of building blocks for VLs that corresponds to NLs’ phonemic (cheiremic) inventory or “alphabet” of sounds (sign structures). Dots, lines, angles?… The list of primitive objects is alarmingly short, compared to NL inventories, and at the very same time some of its members are far too open-ended: an infinite number of distinct lines can be drawn. If our theoretical target is a VL lexicon, built up in the human mind, containing a finite number of discrete graphic entries, we’d better not start with an infinite set of VL graphemes!
38Consider how we might subject Figure 6, for instance, to morphemic analysis. Even though there is nothing here but a dot and a single continuous line, our course is murky. The dot seems to be a minimal graphic unit, and also to be a full-fledged meaning-bearing unit—it’s the eye! This is not necessarily problematic: we might compare meaningful dots to minimal function words like unstressed “a” in English (pronounced /ə/). This sound, /ə/, belongs to the sub-meaningful phonemic inventory of English, but in appropriate environments it is also interpreted as a meaningful word.
39But as for the line, is it a single graphic unit or several? Though it is graphically continuous, it has inflection points. Should it be cut at each one of those? How many meaning-bearing units should it be dissected into? Let’s arbitrarily choose one of the two possible readings of the graphic figure, and say that based on prior narrative cues we know this figure is meant to be a rabbit. We can certainly see it as composed of parts, including rabbit ears, neck, and mouth. How much of the line signifies MOUTH—just the single inflection point? If the neck is a visual morpheme, presumably its component graphemes are two line segments clipped from the bottom of the drawing, plus the white space between them? We will certainly be forced to add “white space” to our graphemic inventory, and the visual morpheme signifying NECK will have to be supplemented with combinatory information about how its components may (optionally) link up with a group of graphemes composing HEAD above and BODY below. Notice that the combinatory properties of continuous graphemes are very different from those of the discrete phonemes and morphemes encountered in NL: a line signifying the meaning RABBIT behaves very differently from a word signifying RABBIT. Pronouncing the sub-sequence /rab/ aloud, for instance, does not get anyone to the meaning RABBIT EAR.
40Nevertheless, we still hold out hope that VLT’s parallel architecture can be repaired, with appropriate adjustments to the graphic modality. We agree that there is absolutely a significant overlap in semantic or conceptual structure between VL and NL even though the combinatory principles are different. And cognitive equipment entailed by the human NL endowment is surely available for other purposes, as the invention of writing attests. Moreover, we are not even arguing that there is no way to construct a theoretical set of graphemic primitives plus combinatory principles allowing the construction of more complex units. Indeed we very much look forward to future proposals regarding VL lexicons and grammar. But whatever such systems might look like, they won’t much resemble the phonology and morphology of natural human language.
41The lack of firm footing at the lowest combinatory levels is no doubt the reason why Groensteen declared the panel the base interpretive unit in Système. To us it seems clear that a semantics of comics has to be willing to formalize units below the panel level. So, even without a solid foundation beneath us, we will simply have to start somewhere in the middle, wherever we can get a toehold. This is the value of figures.
42If there is such a thing as a VL lexicon, we believe its entries on the graphic (signifier) side can only consist of something like figures. Much if not all of the VL lexicon piggybacks on a pre-existing set of concepts already acquired and already matched to phonological content; children have learned what sounds in their native language correspond to the ideas SUN, HOUSE, GIRL, and TREE well before they learn to draw graphic figures to represent the same concepts. In the normal course of childhood events, first a word/concept is acquired, then comes the attempt to represent things and people using marks on the page, then comes the full symbolic abstraction of learning to read and write words. Acquisition of an NL is a near inevitability for any human child; drawing and writing, sadly, are not. Literacy is a marvelous add-on to our human linguistic endowment, but it is wholly parasitical on a pre-existing NL. We leave for a future day any speculation about the relationship between VL and literacy.
43Figures are wordlike, or phraselike, to an extent. Like words, they are reliable pointers to specific conceptual material (that which is signified, diegetic entities). Unlike words, though, the signifier is not discrete, segmentable, easy to digitize and replicate. There’s only one way to spell “tree.” There are always multiple acceptable ways to draw TREE.
44Returning now to higher levels of organization, we confront the question of how figures operate in a narrative sequential context. How many figures does the crucifixion scene (Figure 3) contain? At first blush, we are inclined to say three, although they are not all of equal importance to the narrative. The central figure is more salient than the other two for reasons of both graphic prominence (it’s elevated above the others, facing forward, and we see the entire body) and narrative prominence: this individual is already known to be the protagonist of the story.
45Is the cross not a figure, then? It meets definition 1. For that matter, so do the crown of thorns, the loin cloth, clothing worn by the other two figures, the three nails, the “INRI” label, and the tack affixing the label to the cross. Even the occluded lines in back, representing the gently curving hills or dunes of the geographical setting, cannot be rejected out of hand as potentially figural. After all, the protagonist (so to speak) of Robert Crumb’s “A Short History of America” is a certain intersection in the mid-west which appears in every panel (even before the railroad and roads are built, the place is marked by a natural curvature of the landscape). Precisely the cool thing about the Crumb comic is the reversal of the normal figure-ground relationship between humans and their environment. We want to leave room for our formal apparatus to cope with such outliers.
46In terms of reader attention and narrative momentum, however, “discernible” entities are not all competing on equal ground. Hence, we propose a second definition, that of salient figure, as well as some rules of thumb for assessing salience.
47Definition 2: Salient figures are just those figures parsed by the reader as significant for narrative processing purposes.
48Other things being equal, animate figures are more significant than inanimate, especially if they are anthropomorphic to some degree, and most particularly if they have faces. For Figure 6, we would prefer to say that there is only one salient figure, despite the fact that there are other discernible items within it. What the ears, eye, neck, etc., have in common is that they are proper subparts of a larger discernible entity in the same panel. A rule such as “whole > part” means that we would prefer to read the entire graphic composition in Figure 6 as the salient figure, rather than subdividing it into figural parts. A panel subsequent to Figure 6 certainly might zoom in to just an ear or eye, of course, likely with some additional graphic elaboration, and then that part would be a salient figure. Conversely the next panel might zoom out to show the rabbit’s entire body, in which case the head alone would no longer read as significantly figural.
49Similar tendencies would apply to the interpretation of clothing and some accoutrements. Unless and until there is a narrative reason to distinguish them, we assimilate such entities to the figures who wear/wield them, thus reading “male figure in a cloak” rather than “cloak figure” + “male figure”. (This relationship is akin to that of “head” and “complement” in linguistic phrasal analysis.)
50Characters are more salient if they are repeated/recognizable than if they aren’t, varying along a spectrum that runs from protagonist at one extreme to completely anonymous at the other (e.g., “Roman soldier”). Indeed, in certain crowd scenes, humans may even be reduced to the stature of props. The image foreground can make figures register as more salient, and various formal means of layout can suggest and increase a figure’s salience in particular occurrences. From the higher-level perspective of narrative processing, however, the combinatory (syntactic) features of FS come into play and the main emphasis is on a figure’s reoccurrence as a means of continuity-building and coherence.
- 20 Layout may guide viewing on its own, however, when panels have no content.
51To return to the eye-movement data that we gathered in an earlier study, one key observation that we made was that visual content in the images can guide the readers’ decisions about navigational order relatively independently from layout decisions or text,20 and can also override the impact of layout. Most obviously, the characters’ faces, expressions, gestures, posture, movement, and the repeated depiction of their bodies, invite fixations, saccades, and dwell time. These visual elements must also influence how and what readers see, even if we cannot substantiate this claim directly on empirical grounds. Some elements in the data, nonetheless, also readily suggest the interpretation that FS influences what the readers choose to see and in what order.
- 21 The readers’ interest in this sub-area of particular interest is reflected in our transition matrix (...)
52Let us consider, for instance, the way in which narrative meaning and layout interacted in reading behavior in a wordless scene from Cosey’s Voyage en Italie (1988). (See Figure 8.) Here, we identified a sub-area of panels (AOI 11–14) at the end of the spread as a key area of interest. The readers’ tendency to focus on this area was reflected in a significant number of transitions, regressions, and see-saw patterns, or circling, within these AOIs.21 One of these panels, AOI 13, also attracted the most transitions on the whole spread. The heat map reproduction of the readers’ foci and dwell time further suggests that these panels form a tightly knit unit of readers’ interest that centered around the recurring character.
Figure 8: AOI-manipulated spread from Cosey’s Voyage en Italie, 1988, p. 10–11.
Cosey © Dupuis, Brussels
53This sub-group of panels depicts a narrative scene of continuing action: a girl who has sneaked into an airport cargo area cuts open a corner of a large cardboard box and then takes out a smaller box from that corner. This action raises narrative questions supported and augmented by the layout design: What is she doing? Why is she interested in TWA, New York City, or these boxes? What is the purpose of her action?
- 22 The results of altogether 22 participants were maintained for our research experiment. However, the (...)
54The small key panel (AOI 12) just above AOI 13, which shows in close-up detail the girl’s hand touching a sign that says “TWA New York – USA JFK Airport”, contributes to the same narrative enigma. The panel composition, further, increases narrativity, i.e. the sense of being able to inspire a narrative response (Ryan 2005: 347)—however not in the sense of Neil Cohn’s categories of horizontal stagger or vertical blockage. In terms of layout, AOI 12 is foregrounded in this area to the extent that the panel’s frames break the otherwise even upper line of the lowest panel row. Consequently, the layout arrangement gives the impression that AOI 12 is partly superimposed onto, or perhaps inset into, AOI 10, the panoramic panel above. The arrangement does not, however, violate the Z-path reading pattern to the extent that the path from AOI 10 through AOI 12 to AOI 11 is smooth. More participants also accessed AOI 12 from AOI 10 instead of AOI 11 (15 versus 9).22
55The problem with the route that many readers chose, from AOI 10 to 12 and then to 11, even if it conforms to the Z-path, is that it does not seem to make much narrative sense. More precisely, it is not logical in terms of the evolving narrative scenario: How could the scene of AOI 11, which depicts the girl walking in the cargo area, come after the close-up image that shows her investigating a box in that same space? The continuing action that this sequence depicts is more comprehensible if AOI 11, as a kind of setting for the action, precedes AOI 12 and 13. At the same time, in the close-up image of AOI 12, the character’s hand and the text on the side of the box function as potential cues for the meaning of her action. They indicate the character’s heightened focus, thus raising narrative questions and enhancing the narrative salience of this panel. Thus, we presume, the girl’s evident interest in this box may elicit regressions and “cycling” at the end of the reading as the readers were looking for a potential explanation for her actions. Not just the layout arrangement, then, but also the character’s reoccurrence and continuing action, and the interest in reading for the narrative, may explain the reading order of this passage.
56The readers’ regressions and see-saw patterns in this area indicate the need to understand the intricate correlations and interactions between layout and narrative content, but they also suggest that the comprehension of visual and narrative meanings can override, at least temporarily, the Z-path reading pattern that the layout structure supports. Thus, we hypothesize that the central function of following the character, and understanding the action in which she is engaged from one panel to the next, explains the digressions in this sub-area. The concept of figural solidarity can further help us to identify the readers’ need in this passage to pay two-directional attention forward to subsequent panels and backward to a past occurrence of the same figure.
- 23 See also Mikkonen (2017: 57–60) on the question of the key frame in narrative comics.
57At the same time, we also presume that some panel relations, or groups of connected panels, such as this sub-group of panels, are narratively more salient due to the recurring character engaged in action and/or situation, i.e. they function as key narrative images.23 We hypothesize that narrative salience in comics involves, in particular, recurring and continual characters that may be followed, and whose perspective may also be represented for narrative reasons. One such reason is to develop a sense of coherence in the unfolding story.
- 24 The page turn involves a shift in attention, takes time, and, subsequently, serves both narrative c (...)
58It must be emphasized, moreover, that the meaning of the text on the box in the cargo area as well as the character’s interest in this particular object remain relatively open at this point in the story, and that this sense of openness is further accentuated by the function of page turn as a point of indeterminacy.24 The reader cannot yet know that the girl will go into the cargo box as a stowaway. Only much later in the narrative do we learn that she is a Cambodian refugee who dreams of moving to the United States. Here, on the contrary, other narrative scenarios such as stealing something from the cargo remain plausible. After all, in the last panel the girl is shown walking away from the large box, carrying a smaller box in her hands. We hypothesize, then, that this openness, and the potential suspense that may arise from not knowing what the character is doing, can further increase narrativity in this passage.
59Finally, we want to return to the complex question of why readers focus on characters. This seems to happen often for several reasons at the same time. Readers keep track of characters to develop their understanding of them, to comprehend what the characters feel and think, or what they intend to do, thus creating and updating a mental model of them. Yet, since character, action and situation are inseparable, readers also follow characters to make sense of a narrative event, plot, and the story’s overall meaning. Continuing visual content helps the reader to build meaning representations that are coherent, and can make the movements between the panels smooth, while a recurring character’s salience may also require digressions from conventional reading order.
Flux RSS
