Skip to navigation – Site map

HomeNuméros7Dossier3. Transpositions between verbal ...Intersemiotic Translation as Rese...

3. Transpositions between verbal and non-verbal semiotics

Intersemiotic Translation as Resemiotisation: A Multimodal Perspective

Kay L. O’Halloran, Sabine Tan and Peter Wignell
p. 199-229


Intersemiotic translation is viewed as the basis of cultural communication through which ideas are circulated, translated and explained using language, images and other semiotic resources. From this perspective, intersemiotic translation is conceptualised through the principle of resemiotisation which is concerned with how semiotic choices are translated over time as social practices. Key issues which arise from viewing intersemiotic translation as resemiotisation are investigated in this paper: (a) how can shifts of meaning be conceptualised across semiotic resources which are fundamentally different in nature?; (b) what meanings are retained and changed as a result of resemiotisation?; and (c) how can such meanings be modelled theoretically and tracked analytically? In order to explore these issues, a multimodal social semiotic approach based on Michael Halliday’s systemic functional theory is presented. In this approach, semiotic resources are conceptualised as systems of meaning which differentially fulfil various functions in society: to structure experience and make logical connections in the world, to enact social relations and create a stance towards the world, and to organise these meanings into multimodal messages. The multimodal approach to intersemiotic translation is illustrated through examples which explore shifts of meaning which take place across theses strands of meaning as semiotic choices are resemiotised within and across multimodal texts. The focus of this discussion is intersemiotic translation across language (spoken and written), images (photographs, film tracks and graphs) and mathematical symbolism in a range of different text types (i.e. a news report, an infographic, a video and a mathematical graph). These examples illustrate the principles of intersemiotic translation through which semantic expansions occur as semiotic choices are resemiotised using different semiotic resources, which result in access to a new meaning potential.

Top of page

Full text

1. Introduction to intersemiotic translation

1Translation is typically thought of as involving language, in particular written language. However, translation extends beyond linguistic translation and the interpretation of linguistic signs through rewordings to “intersemiotic translation”, which, in its original sense, was seen as involving the interpretation of linguistic signs by means of non-verbal resources (Jakobson 1959, p. 114). Jakobson (Ibid., p. 233) distinguishes three ways of interpreting verbal signs: intralingual translation (translation into other signs of the same language); interlingual translation (translation into another language); and intersemiotic translation (translation from language into another, nonverbal system of symbols). For Jakobson, intersemiotic translation involved language. He does not discuss translation from one non-verbal semiotic system to another non-verbal semiotic system or the translation of multisemiotic texts.

2Today, Jakobson’s (1959) definition of intersemiotic translation has been broadened to include translations across non-linguistic semiotic resources (e.g. Kourdis and Yoka, 2014). This development seems inevitable, given the proliferation of different forms of multimodal texts in today’s digital environment, where semiotic resources (e.g. language, image and sound resources) “coexist, cooperate, and get translated” (Kourdis 2015, p. 311) on a regular basis. In this regard, intersemiotic translation, the constant translation of signs into other signs, forms the basis of cultural communication, as Eco (1979) explains:

[…] culture continuously translates signs into other signs, and definitions into other definitions, words into icons, icons into ostensive signs, ostensive signs into new definitions, new definitions into propositional functions, propositional functions into exemplifying sentences, and so on; in this way it proposes to its members an uninterrupted chain of cultural units composing other cultural units, and thus translating and explaining them. (Eco 1979, p. 71)

3Eco’s (1979) view of intersemiotic translation as the basis for cultural communication is adopted here. From this perspective, intersemiotic translation is conceptualised as “resemiotisation”, which is concerned with “how semiotics are translated from one into the other as social processes unfold” (Iedema 2003, p. 29). Key issues which arise from viewing intersemiotic translation as resemiotisation include: (a) how can shifts of meaning be conceptualised across semiotic resources which are fundamentally different in nature?; (b) what meanings are retained and changed as a result of resemiotisation?; and (c) how can such meanings be modelled theoretically and tracked analytically?.

4The phenomenon of intersemiotic translation remains “virtually unexplored in terms of its conceptual modelling, especially from a semiotic perspective” (Aguiar and Queiroz 2009, p. 1) despite it being theoretically relevant and commonly practised. While Jakobson’s view of intersemiotic translation always included language, Aguiar and Queiroz (Ibid., p. 1) extend the principle to include “translations of texts of all kinds”. Aguiar and Queiroz also propose that the main methodological difficulty in intersemiotic translation is the comparison of different semiotic resources, and they critique current models as being mainly descriptive, lacking explicative models and being “dissociated from results produced in the area of general semiotic and translation studies” (Ibid., p. 7). They attempt to resolve this methodological difficulty by adopting a conceptual framework based on Peirce’s model of sign process, as a starting point (Ibid., p. 2).

5Perez-Gonzalez (2014) cites conflicting views of intersemiotic translation in the translation literature, with some scholars (e.g. Gottlieb 1997, p. 111; Remael 2001, pp. 13-14) regarding intersemiotic translation as the transfer of meaning across different media, while others (e.g. Fine 1984) understand it as shifts between two different variants of the same sign system, such as changing from spoken to written language in film subtitling. Perez-Gonzalez (2014) also cites a lack of consensus on the definition of terms such medium, mode, and sign system, which “exposes the need for a more comprehensive and sophisticated understanding of the semiotic fabric of translated and interpreted texts” (Ibid., p. 120).

6O’Sullivan (2013, p. 5) notes that translation studies has “struggled at times with the concept of multimodality”, which is concerned with the study of the various semiotic resources involved in communication (see different approaches to multimodality in Jewitt, Bezemer and O’Halloran 2016). O’Sullivan (2013) explains that while some progress has been made in developing methodologies for research on the translation of multimodal texts, a number of technical and logistic challenges remain (Ibid., p. 6). She goes on to say that “it makes sense then that the saturated multimodality of many texts today would require both a new, or at least a rethought, critical and analytical toolbox, and potentially also new approaches to translation” (Ibid., p. 6).

7At least some of the difficulties in developing a theoretical framework for intersemiotic translation arise from attempting to find a “direct” translation between meanings made through choices from semiotic systems that are fundamentally different in nature (e.g. language and image). As Iedema (2003, p. 47) points out, translations between different semiotic resources inevitably introduce discrepancies. While an exact intersemiotic rendition is highly improbable, if not impossible (except in certain cases, see discussion of mathematics below), a reasonable approximation is certainly likely. The sticking point, as O’Sullivan points out, is in finding “a critical and analytical toolbox” (O’Sullivan 2013, p. 6) which is capable of theorising and modelling the shifts of meaning which take place through intersemiotic translation.

8In this paper, intersemiotic translation is conceptualised through the principle of resemiotisation, which is concerned with “how meaning shifts from context to context, from practice to practice, or from one stage of a practice to the next” (Iedema 2003, p. 41). Iedema first used the term to account for “the origin and dynamic emergence” of those shifts in meaning (Ibid., p. 40). Jewitt (2014, p. 467) summarises Iedema’s (2003) use of the term resemiotisation as the phenomenon “in which a particular set of meanings is transformed from one semiotic system (and configuration of media and modes) to another as social processes unfold”. The examples Iedema (2003) cites illustrate changes in context over time and trace the stages in a social process, such as how what is said and agreed on at meetings is summarised into a written report, which is then resemiotised as an architectural design. This view of resemiotisation implies successive stages over time but it also implies multimodality. Each stage involves taking meanings from a source(s) and reconstructing them in a different mode or medium, which itself might be multisemiotic.

9While Iedema (2003) focuses on the processes or dynamics of resemiotisation, it is also possible and productive to examine the actual products of resemiotisation (i.e. the multimodal texts) which are circulated and resemiotised by members of a culture. For people who did not have access to the process, the products are the basis of cultural communication through which social processes are (re-) configured. From this perspective, the view of intersemiotic translation adopted in this paper is that any attempt at translation of meanings made though choices in one semiotic resource into meanings made though choices made from another semiotic resources involves resemiotisation in terms of process and product. In this paper, the focus is intersemiotic translations which take place within and across the semiotic products or artefacts resulting from resemiotisation processes: that is, the focus is intersemiotic translation in the multimodal texts themselves.

10To address the issues discussed above, the aim of this paper is to present a multimodal approach to intersemiotic translation which provides a common theoretical platform for conceptualising different semiotic resources and exploring the shifts of meaning which take place through resemiotisation. The approach is based on Michael Halliday’s (1978) social semiotics, where language and other semiotic resources are viewed as resources for making meaning. Halliday’s (1978) social semiotic approach, developed as systemic functional theory (SFT), views semiotic resources as systems of meaning which fulfil a range of functions in human communication. The systems of meaning are unique to each semiotic resource, but configurations of system choices work together in order to construct thought and reality in specific ways in any culture. This theoretical approach is discussed in detail in Section 3.

11Before introducing the systemic functional approach to intersemiotic translation, other semiotic and multimodal approaches to intersemiotic translation are reviewed. Following this, the basics of Halliday’s systemic functional theory (SFT) are presented. The multimodal approach to intersemiotic translation is then developed and illustrated through examples which explore shifts of meaning which take place as semiotic choices are resemiotised. The focus of this discussion is language (spoken and written), images (photographs, graphs and a film track) and mathematical symbolism in a range of different text types (i.e. a news report, an infographic, a video and mathematical graphs) found on an Internet website, in this case, the World Health Organization Ebola website.1 The wide range of examples chosen demonstrates that the systemic functional approach can be applied to different types of intersemiotic processes, which in this case, occur within the context of a single website. The examples analysed below also demonstrate that new methods for investigating intersemiotic translation are required: in this case, purpose-built software applications for text, image and video analysis are demonstrated to handle the complexity and multi-level nature of multimodal semiosis.

2. Multimodal approaches to intersemiotic translation

12Many text types involve language in combination with other semiotic resources, such as still and moving images, diagrams, graphs, music and typography. Multimodality can be defined as “the use of several semiotic modes [i.e. resources] in the design of a semiotic product or event” (Kress and van Leeuwen 2001, p. 20, cited in Jewitt 2014, p. 1). Multimodal texts involve more than language. They can include meanings made from choices from any semiotic system and do not necessarily need to incorporate language. Any text which utilises more than one semiotic resource is a multimodal text.

13O’Sullivan (2013, p. 2) points out that multimodal meaning-making is deployed for many purposes, such as promotional, political, expressive and informative purposes and that signifying elements other than language must be understood and accounted for by professionals who work with text, such as technical translators, literary translators, copywriters, subtitlers and publishers. The list could be expanded to include people who work with the design and creation of multimodal texts, such as web designers and graphic designers. O’Sullivan also points out that, while multimodality has long been present in texts, it “has become increasingly conspicuous” with the development of the World Wide Web and new forms of communication and entertainment (Ibid., p. 5).

14Taylor (2013, p. 98) comments on the growing importance of multimodality, especially for audiovisual translation, and says that “a limited number of scholars have in fact ventured into this field”. Taylor (2013) also comments that “the role of translation in multimodality studies … seems to have attracted little or no interest” from outside the field of translation. Taylor (2013) does cite the work of a number of scholars influenced to varying degrees by Halliday’s (1978) ideas on language as a social semiotic as opening doors “to a growing interest in how different modes worked together to create meaningful texts” (Taylor 2013, p. 98).

15One such study is Borodo’s (2015) multimodal analysis of the relationship between the verbal and the visual in the translation of comic books. Borodo’s (2015) work focuses on how the verbal and visual modes interact and jointly contribute to creating meaning and is influenced by the work of Kress and van Leeuwen (1996, 2001), Jewitt (2014), Kress (2009) and Royce (2007). Borodo (2015, p. 23) argues that language is one element within a larger semiotic framework that includes images, gesture, posture, gaze and colour, which should be viewed as possessing equal (but not equivalent) meaning-making potential. Borodo (2015, p. 40) proposes that “investigating comics from a multimodal perspective may be another step towards a more complete understanding of the nature of this still largely unexplored sphere within Translation Studies”.

16The assumption so far appears to be that intersemiotic translation involves translation between texts, taking meanings from one text and transposing them as accurately as possible into another text. While this is probably the most important aspect of intersemiotic translation for professional translators, there is also the phenomenon of intersemiotic translation within a text to consider. That is, in the same text meanings encoded by one semiotic resource are often re-encoded, or resemiotised, through another semiotic resource: for example, information in a graph could be re-expressed in language or a photograph could be resemiotised as an infographic. Phenomena such as these are important for people designing and creating these texts and for the people who read and view them. Beyond this, as intersemiotic translation forms the basis for explaining and circulating ideas in society and culture, it has implications that extend well beyond translation and multimodal text design.

3. A multimodal systemic functional approach to intersemiotic translation

17The approach to intersemiotic translation adopted here is derived from multimodal social semiotic theory, based on Halliday’s systemic functional theory (SFT). In SFT, semiotic resources are conceptualised as inter-related systems which together constitute and manifest culture (Halliday, 1978, 2008; Halliday and Hasan, 1985). While Halliday studied language, he always understood that language was one semiotic resource among the many (e.g. images, gesture, dress and sounds) which constitute culture and the fundamental principles of the approach are applicable for the study of other semiotic resources. The view of multimodal semiosis, in which recognizable configurations of semiotic choices constitute social practices, provides the basic foundations of the approach. Significantly, the SFT approach is capable of handling the multi-dimensional complexity of intersemiotic translation as resemiotisation, as explained below.

18Semiotic resources are viewed as having a meaning potential which is described in terms of interconnected systems of meaning. Systemic functional approaches to multimodal discourse analysis (SF-MDA) (e.g. see Chapter 3 in Jewitt, Bezemer and O’Halloran 2016; O’Halloran, Tan and Marissa 2015a; Tan, Smith and O’Halloran 2015) are concerned with the “grammatics” of semiotic resources, which involves formulating the systems of meaning in order to understand the functions of different semiotic resources and the meanings which arise when semiotic choices combine in multimodal phenomena over space and time (O’Halloran and Lim 2014). Following Halliday’s SFT, the systems are organised according to the functions (called “‘metafunctions”) which the resources serve in society (e.g. Halliday and Matthiessen 2014; Martin and Rose 2007):

  • Ideational meaning which consists of experiential meaning: to structure experience of the world and logical meaning: to make logical connections in the world;

  • Interpersonal meaning: to enact social relations and create a stance towards the world;

  • Textual meaning: to organise experiential, logical and interpersonal meanings into messages

19The meanings of multimodal processes and texts are characterised in terms of options selected from the systems which realise the metafunctions (i.e. ideational, interpersonal and textual). For example, systems of meaning have been formulated for language (Halliday and Matthiessen, 2014), static images (Kress and van Leeuwen, 2006; O’Toole, 2011), music (van Leeuwen 1999), action (Martinec 2001, 2004) and film resources (Bateman and Schmidt 2012). The systems are typically organised according to different ranks of constituency (e.g. discourse semantics, lexico-grammar and phonology/graphology for language; and work, episode and figure for image) (see Section 4 below).

20While the metafunctional principle that each semiotic resource is organised to realise four different strands of meaning simultaneously (i.e. experiential, logical, interpersonal and textual) is applied to any semiotic system, the underlying systems and choices are different for each semiotic resource. That is, semiotic resources have differential capacities with regards to the different metafunctions and the subsequent meanings which are made. Images, for instance, do not structure and order the world in the same way as language does. Images order human experience by situating happenings in relation to other happenings, as parts of a whole, unlike language where happenings are constructed in sequential order. For example, in a photograph, painting or a scientific diagram many happenings and actions are taking place in relation to each other simultaneously. However, certain aspects of the image are made salient through semiotic choices such as gaze, light, colour and framing and immediate features of the context of the situation (e.g. instructions to view parts of the image, or captions). That is, images are ‘read’ in particular ways, depending on the semiotic choices which are made within the image and the context. In this regard, multimodal semiosis results in an expanded meaning potential derived from the integration of different metafunctional capabilities. As illustrated by the examples considered below, intersemiotic translations permit semantic expansions which extend beyond those possible with one resource alone.

21The metafunctional principle plays an important role in SF-MDA for understanding the functionalities and underlying organisation of semiotic resources and investigating the ways in which semiotic choices interact to create meaning. As O’Halloran and Lim (2014, p. 140) explain, the metafunctional organisation of meanings is particularly useful for SF-MDA,

[…] because it provides a common set of fundamental principles to compare semiotic resources and the meanings which arise when semiotic choices integrate in multimodal text. That is, the organisation of metafunctional meanings offers a unifying platform for studying semiotic resources and their inter-semiotic relations. (O’Halloran and Lim 2014, p. 140)

22In SF-MDA, the focus on the metafunctional organisation of semiotic resources is critical, but the actual choices in multimodal texts and processes are interpreted in relation to the context using the concepts of register and genre (e.g. Eggins 2004; Martin 1992, 2002; Martin and White 2005). That is, choices from multimodal systems of meaning form more or less stable (but evolving) configurations, which are socially and culturally recognisable. That is, while the meaning potentials of semiotic resources are diverse, the actual options selected in any context are conditioned by previous choices within that culture. These dimensions are described using register theory (Halliday 2002 [1977]; Matthiessen 2009, Martin, 1992, Martin and Rose, 2003), which is concerned with three key dimensions: field—the nature of the social activity (realised through experiential and logical choices); tenor—the social relations which are enacted (realised through interpersonal choices); and the mode—spoken, written and visual forms of representation (realised through textual choices). The genres found in any culture are realised through the various configurations of register variables (i.e. field, tenor and mode). In this case, genre is defined as “the system of staged goal‑oriented social processes through which social subjects in a given culture live their lives” (Martin 2002, p. 56). Therefore, while multimodal semiosis “multiplies” the potential meanings which can be made, in reality meaning is constrained according to context and culture (e.g. Lemke 1998).

23The multi-level systemic model, in which semiotic resources in multimodal texts are theorised in terms of metafunctionally organised systems of meaning and semiotic choices are analysed according to register and genre, opens up space for an SF-MDA approach to be applied in order to trace the shifts in meaning which take place through intersemiotic translation. That is, the application of a unified theoretical model which applies equally well to different semiotic resources permits the effects of intersemiotic translation to be calibrated, as illustrated in the examples which follow.

4. Examples of SF-MDA approach applied to intersemiotic translation

24The SF-MDA approach to intersemiotic translation is demonstrated by analysing the meanings which arise in a news report with linguistic text and photographs, an infographic, a video and a mathematical graph. In each case, intersemiotic translation is explored using the concepts of metafunctionally organised systems of meaning, register and genre. The systems are organised according to different ranks of constituency for each resource. For example, the text and image systems in Table 1 are organised according to discourse semantics, lexico-grammar and phonology/graphology for language; and work, episode and figure for image. These systems are based on Halliday’s (Halliday and Matthiessen 2014) and Martin’s (Martin and Rose 2007) systems for language and O’Toole’s (2011) framework for images. Other systems are also considered, for example, cinematography in video and film (see Table 2) and mathematical symbolism (see Table 3). In what follows, the text and image systems in Table 1 are used to analyse the intersemiotic translation of meanings between text and photograph in a new report from the WHO Ebola website. Following this, intersemiotic translations which occur in a video and a mathematical graph from the same website are explored.

25In order to handle the complexity of such analysis, purpose-built software tools, Multimodal Analysis Image2 and Multimodal Analysis Video3, which are specifically designed to explore semiotic interactions in static (e.g. written texts and images) and dynamic media (e.g. videos) respectively, are used. The software applications permit media files to be imported and analysed using different systems which are entered into the software, and the results are stored in a database for further data processing (see O’Halloran, Tan and Marissa 2015b).

Table 1








Text & Discourse semantics

Ideation & Taxonomic relations

Organizing experience into activity sequences and class/sub-class and whole/part relations


Narrative Theme; Representation; Setting

Nature of the scene

Clause (lexico-grammar)

Processes; Participant Roles; Circumstance

Happenings, actions and relations


Processes; Participant Roles; and Circumstance

Visual happenings, actions and relations


Posture; Dress

Characteristics of the participants


Text & Discourse semantics

Exchange Structure
Speech Function

Exchange of information (e.g. statements and questions) and goods & services (e.g. commands and offers)


Angle; Camera Distance; Lighting

Visual effects

Clause (lexico-grammar)


Declarative or interrogative (for exchange of information)
Imperative and modulated interrogative (for exchange of goods and services)


Proportion in Relation to the Whole Image: Focus; Perspective

Happenings, actions and relations with respect to the whole image


Gaze-Visual Address

Direction of participant’s gaze as internal to image or external to viewer


Text & Discourse semantics

Reference and Retrieval

Identifying and tracking participants in text


Compositional Vectors; Framing

The organisation of the parts as a whole, with the visual marking (e.g. framing) of certain parts

Clause (lexico-grammar)

Information Focus

Organisation of information, with points of departure for what follows


Relative Placement of Episode; Framing

Position of the happenings, actions and relations in relation to the whole image, and the visual marking of certain aspects


Relative Placement of the Figure within the Episode; Arrangement; Framing

Position of figures in relation to happenings, actions or relations, and the visual marking of certain aspects of those figures

Text and image systems

4.1. Intersemiotic translation between photograph and text

26This example discusses intersemiotic translation between two photographs and accompanying text in a news story “Ebola—a test too far for one little girl”4 from the WHO Ebola website displayed in Fig. 1a. Fig. 1b shows an excerpt of the story as analysed in Multimodal Analysis Image software, using the text and image systems displayed in Table 1. The story has been selected to demonstrate the principles of intersemiosis and resemiotisation between the linguistic text and the photographs. The section of the story used for illustrative purposes consists of both photographs and the first four paragraphs of linguistic text following the introductory paragraph, displayed in Fig. 1b. The photographs and linguistic text realise ideational, interpersonal and textual choices, where the choices are made from different sets of options from different systems, as displayed in Table 1. The results of these choices, when combined, realise meanings which are different from those realised by each semiotic resource. This intersemiosis is multi-directional. Whether the text is read after viewing the photograph or the photograph is viewed after reading the text, the meanings which can be made from each, both individually and together, are different from the meanings which are made by each resource in isolation. This transference of meaning is also not a single occurrence. Each time a viewer/reader moves from image to text or the other way around, something new emerges that changes how both are perceived.

27The photographs show two groups of people involved in discussion or negotiation. From the photographs alone a viewer cannot tell what the people are talking about or what relationships there are among them. For the purposes of this discussion, it is useful to start with textual or compositional meaning. Textual meaning is organised differently in the linguistic text and in the photographs.

Figure 1a

Figure 1a

News story ‘Ebola – a test too far for one little girl’4 as displayed on the WHO Ebola website

28In linguistic texts, information flows from the beginning to the end of the text in waves of different scales (Martin and Rose 2007 pp. 175-205) and the position of that information tells the reader whether or not it is prominent as thematic or new information. In the photograph all the information is presented simultaneously and different resources such as Foregrounding, Parallelism and Relative Position and Proportion in the work as a whole are used to identify points of prominence (O’Toole 2011, pp. 23-29). For example, in both photographs compositional choices guide the viewer to see the man in the white T-shirt and dark sleeveless jacket as prominent (outlined in red in the photographs in Fig. 1b). He is the focus of the other people’s attention. Even though he is not foregrounded in either photograph he is still textually the most prominent figure in both. While his Gaze-Visual Address is directed at what he is holding in his hands and he is absorbed in a telephone conversation in the first photograph, the gaze vectors of the other participants are directed at him (indicated by blue directional arrows in Fig. 1b). These gaze vectors draw the viewer’s attention to him even though he is not in the foreground. In the second photograph the upper torso and head of the same man are framed by the window behind. This draws attention to the parts of him that are framed. He is also facing the viewer in both photographs while most of the other participants have their backs to the viewer.

29The same man is also made thematically prominent in the parts of the linguistic text that relate most directly to the photographs. The man is identified in the text as Dr M’Bemba Camara and as “a member of WHO’s surveillance team in Guinea”. By name and by role he has thematic prominence in three clauses in the first paragraph following the story’s introductory paragraph. In the linguistic text discourse semantic resources of reference and retrieval are deployed to establish Dr M’Bemba’s identity and track him through pronouns. As well as being identified by name, Dr M’Bemba is identified as him, he and his five times in the four paragraphs (outlined in shades of red in the text in Fig. 1b). In the photographs the viewer can identify him visually. As the photographs depict episodes in a single instance of time and he appears once in each photograph, it is relatively simple to track his identity across the two photographs. By combining information in the text and photographs the viewer/reader can specifically identify who Dr M’Bemba is among the participants in the photographs. This is done largely through his clothing in both photographs and through the medical apparatus (the contactless thermometer and checklist) he is holding in the second photograph. Of the other participants in the photographs four are identified. The girl, Mariam, is identified by name and can be identified in the second photograph by inference since she is the only girl in the photograph. The other three can be identified by inference and by drawing connections between the photographs and the linguistic text. The older man, wearing the cap in the first photograph and seated in the second photograph is, by inference, Mariam’s father. The viewer can also infer, less strongly, that the woman in the first photograph is Mariam’s mother. (She could also be part of WHO’s surveillance team.) The other male participant in the first photograph is most likely to be the social mobilizer mentioned in the story. The roles of the other two male participants in the second photograph cannot be determined. Choices from two different systems of semiotic resources combine to both identify him and reinforce his prominence. Prominence of participants in the photographs tends to parallel and reinforce prominence in in the linguistic text and vice versa.

Figure 1b

Figure 1b

Intersemiotic translation between photograph and text as analysed in Multimodal Analysis Image software

30Interpersonally, at the rank of discourse semantics, the photographs and linguistic text both function to present information to the reader/viewer. In the linguistic text the whole text grammatically is in declarative mood, the default mood choice for the speech function of giving information. In terms of the register variable of tenor, the linguistic text establishes Dr M’Bemba’s professional and institutional status because he is a doctor and he is working for WHO. This status is reinforced in the photographs where he is the focus of the gaze vectors of all the other participants while in each photograph he is looking elsewhere. The function of the photographs to present the viewer/reader with information is also reinforced by the absence of any gaze from within the photographs directed externally towards the viewer. There are no overt choices made to engage the viewer, who is positioned as an observer of the scene.

31Experientially, the photographs and linguistic text complement and reinforce each other. Both photographs are also primarily experiential: they present episodes as they were in life. At the rank of genre, while the text is a news report, its generic structure is that of a narrative (e.g. Eggins 2004). The photographs represent two stages of that narrative (the orientation and the complication): the orientation, Mariam’s case coming to the doctor’s attention; and the complication, the doctor reporting his conclusions and recommendations to Mariam’s father. The rest of the narrative is carried by the linguistic text. The experiential information contained in each photograph is situated at the rank of episode. The language related to these episodes is encoded in clause complexes and conjunctively related clause complexes. What happens at the rank of episode in the photograph aligns more or less with what happens in clause complexes and in larger pieces of text.

32There are parallels between what happens at the rank of figure in the photograph and at clause rank in the story. In the linguistic processes unfold one at a time and one per clause as the text develops. For example the doctor is the actor in a number of material processes (verbs shaded in red in the text in Fig. 1b): he has been “working”, he “sets out”, he “arrives”, he “begins”. He is also the receiver in verbal processes: people “bring” him information the monitoring team “tell” him things and he “is” also well accepted. Other participants in the text also engage in different processes. The key point here is that, in the linguistic text, participants can only engage in one process per clause: one process at a time, in a series of happenings which unfold. On the other hand, in a photograph the same person or thing can be a participant in a number of processes simultaneously. For instance, in the second photograph, Dr M’Bemba is simultaneously engaged as the ‘doer’ in looking, standing, holding and probably speaking. He is also the ‘done to’ in that he is being looked at and listened to. The key difference between the visual action and linguistic action is that, in photographs, participants can be engaged in multiple processes simultaneously, although the images are viewed in particular ways, given the compositional and interpersonal choices as illustrated here. In this regard, a key function of language is to explicitly order happenings in the world as a logically connected series of events, as highlighted in this example. In the photographs and in the section of the linguistic text analysed, however, Dr M’Bemba is the main participant. He engages as the in more processes than the other participants and he is the focus of the actions of other processes.

33In summary the combinations of choices across metafunctions and across language and photograph build and reinforce the position of Dr M’Bemba, and by extension WHO as the central participant in the whole text. Linguistically, at the rank of genre, the narrative is used as an illustration of WHO’s role in combatting Ebola. Dr M’Bemba, as a representative of WHO, is highlighted experientially as the central participant in that narrative. Likewise, the photographs embedded in the narrative highlight Dr M’Bemba (and WHO) as the major participant. Interpersonally Dr M’Bemba is shown as having status (as a doctor) and authority (as a representative of WHO). Textually he is also highlighted through thematic choices in language and through compositional choices in the photographs.

34The discussion above has demonstrated that the principle of metafunctions can be applied to sets of choices from different semiotic systems to show that, although the choices are different, they realise complementary and compounding meanings. Choices made from one set of semiotic resources reinforce metafunctionally aligned choices made from a different set of semiotic resources, resulting in semantic expansions which extend beyond those possible with either language or image alone. The resemiotisations have the effect of, by using a specific example, highlighting the role of WHO in fighting the Ebola epidemic.

4.2. Resemiotisation in an infographic

35The same principle can be illustrated through discussion of resemiotisation in another multisemiotic text. Fig. 2 is an infographic from the home page of the WHO Ebola website. The infographic exemplifies resemiotisation between language and graphics. The linguistic text and the graphics appear simultaneously on the website as one multimodal text. The focus of the discussion is directed towards resemiotisation through the textual and experiential metafunctions.

36The infographic consists of five separate parts, each of which is concerned with a different aspect of Ebola. The parts are separated from each other by bold horizontal lines and by a straight blank space vertically down the centre. Despite being separated they are also unified compositionally by having much the same general layout of a bold, large heading in the upper left, smaller supporting text and prominent, stylised images which use the same colours and basic shapes.

37Textually, the position of the headings suggests a reading/viewing path based on written English. These headings are thematic and serve as the topic for their respective parts. The accompanying text presents new information related to its topic. Each image resemiotises some of the information in the written text, reinforcing and highlighting its function as salient new information. The combination of linguistic elements and images in the composition of the individual parts and in all of the parts together creates a visual cohesion which assists the reader/viewer in following the flow of information.

Figure 2

Figure 2

Resemiotisation in an infographic

38Some of the features noted in the textual organisation of the infographic are also echoed in its ideational organisation. Notably, participants in the main processes in the language of the infographic are also encoded in the images. For example, in the top left part, the text lists “headache”, “fever” and “vomiting” among the symptoms of Ebola. The symptoms also share taxonomic lexical relations as co-hyponyms and as hyponyms of “symptoms”. Three of these symptoms (“headache”, “fever” and “vomiting”) are resemiotised as stylised, but quite congruent, images (see Fig. 3). In this way, entities in the linguistic text (i.e. “headache”, “fever” and “vomiting”) are intersemiotically translated into visual processes with human participants as well as a relational process with a material object (thermometer). This intersemiosis creates a visual connection from the abstract entities in the heading and written text to the concrete material world of human life which is visually depicted in order to explain how the effects of the Ebola virus can be identified.

Figure 3

Figure 3

Ebola symptoms resemiotised as images

39The other parts in the larger infographic also resemiotise linguistic participants as visual processes and participants. This creates a logico-semantic relationship where the meaning of the images depends on language: that is, the written text in the infographic is necessary to interpret the images. The text, on the other hand, would still make sense without the images but its message would be much more abstract without the accompanying images. However, unlike in the photographs, where the images depict processes and participants in real life, the images in the graphics are highly stylised and experientially pared down to the bare minimum required to be unambiguous for the purpose of providing information to the reader/viewer.

4.3. Resemiotisation between different media

40This example demonstrates how different semiotic resources have been chosen to realise similar meanings in two different media using visual resources only. The focus here is on the resemiotisation of experiential and interpersonal meaning in examples are taken from different parts of the WHO Ebola website as they appeared on 22 June 2015.


Figure 4

Figure 4

Resemiotisation between different media – graphic and photograph

42The stylised image of the three figures in Fig. 4 with their arms raised (left image) appears in an infographic and the photograph (right image) is from the main visual display of the website and also from a news story which is linked to the main visual display. The photograph has been cropped to show just the three central figures. In both images, the experiential content is similar in terms of the gender and age balance. In the photograph the gender and relative ages of the participants is clear, because they are shown as they appeared in real life. This option is not available to the designer of the infographic so other options are chosen in terms of stylised dress and relative size. Compositionally the figures in the photograph and in the infographic are similarly placed, with the adult male in the centre, the female figure to his right and the smaller male figure to his left, although the larger male figure is more foregrounded in the infographic. The foregrounding emphasises the relatively larger size of the male figure.

43Interpersonal meaning is also resemiotised across the two images. The figures in the photograph are facing the viewer, as are the figures in the infographic. The gaze of the participants in the photograph is directed at the viewer. While the viewer cannot see the faces of the figures in the infographic, the white space under their heads indicates chins, which would only be visible if they were facing the viewer. The figures in the photograph are all smiling, showing positive affect. Happiness cannot be shown in the same way in figures without faces so alternative choices are used. In the infographic the figures appear to be jumping for joy: they have their arms upwards and outwards in symbolic happy gestures. A further indicator of happiness can be seen in the blue cloud shapes above the three figures in the infographic. The colour blue here symbolises blue sky, a typical sign of positive affect, even though it is the clouds which are blue rather than the sky. The combination of blue sky and positions of the arms realises a similar interpersonal effect through different sets of choices.

Figure 5

Figure 5

Resemiotisation between different media – graphic and video

44A similar intersemiotic translation can also be found in the connection between the stylised image of disembodied hands washing in the infographic and the still frame of the WHO video entitled “Hand hygiene in Ebola care facilities”, as featured under “Latest videos” in the “News” section of the web page, which shows a woman washing her hands (Fig. 5). Here the same action is replicated in two different media. The graphic highlights the hands by showing them just as hands, with no body attached. Blue coloured circles representing soap bubbles create a contrast with the solid black of the hands. The simple graphic is minimalist with respect to the experiential information required to convey the message. The video still frame, on the other hand, is realistic, showing a woman washing her hands in real time. The translation between the two depends on the purpose of what is being translated. For example, the graphic shows only what is crucial to the act of hand-washing, hands and something to wash them with, so the translation focuses only on those essential elements. The video, on the other hand shows the whole process of hand-washing in moving images, with narration, captions and background conversation, as discussed in the next example.

4.4. Resemiotisation in a video

45Tracking intersemiotic relations in dynamic media, such as films and videos, for example, which “incorporate both visual and acoustic modes of signification, as well as different graphic sign systems” (O’Sullivan 2013, p. 6), is inherently more complex than unpacking the intersemiotic relations between text and images. In order to facilitate the transcription of multi-semiotic resources in audio-visual texts, Taylor (2003, 2013) proposes a page-based multimodal transcription technique that involves inserting images of still frames and meta‑textual descriptions into a table (with rows and columns), so as to establish “where meaning was being created in a multimodal text…” (Taylor 2013, p. 102). Another way of handling this complexity is using software such as Multimodal Analysis Video (O’Halloran and Lim 2014; O’Halloran, Tan and Marissa 2015b), which includes facilities for importing and organizing video files; creating and editing catalogues of system frameworks and system choices for video annotation; storing and consolidating projects of analyses; annotating and analysing videos by creating time-stamped annotations; visualizing combinations of multimodal choices; and exporting data from the analyses to Excel spread sheets for further data processing and visualisation. In this case, the analysis of the WHO video “Hand hygiene in Ebola care facilities” performed with Multimodal Analysis Video software, aids in the identification of intersemiotic relations and transformations that occur as information is resemiotised across on-screen captions, voice-over narration, and the WHO official’s internal dialogue. The systems used for the analysis of the video are displayed in Table 2.

46Table 2 shows that, in dynamic texts, combinations of semiotic choices work together to realize multiple meanings simultaneously (although one or the other metafunction might predominate at a given time), resulting in constant shifts of meanings as the video unfolds (see also Figs. 7 and 8). The analysis reveals, for instance, that textually, the on-screen caption “Hand hygiene in Ebola care facilities” (Fig. 6, left) is the first caption that viewers see when they watch the video. Indeed, it is similar in wording to the title shown in the still-frame featured under “Latest videos” in the “News” section of the Ebola webpage (see Fig. 5). Although probably added last in terms of post-production processes, the initial caption functions like a title in an introductory phase that provides the context for the whole video. This function also appears to be manifested visually, as it is displayed in a much larger font-size than the rest of the on-screen captions (see Fig. 6). The on-screen captions clearly fulfil different functions: the first on-screen caption summarises the whole video, whereas the other on-screen captions only resemiotise certain parts of the WHO official’s dialogue, as discussed below.

Table 2

Metafunctional Orientation

Semiotic Resource & Rank



Experiential Meaning


Register (Field)

Processes, Participant Roles

The experiential component of Register describes the way language is used in context, i.e. what the text is all about, what is happening (Field).

Interpersonal Meaning

Discourse Semantics

Speech Function

Linguistically, verbal interactions fulfil four primary speech functions for exchanging information (e.g. statements, questions), or goods and services (e.g. commands, offers).

Textual Meaning

Dialogic Space (spoken)


Depicts how much time (and space) is allocated to the verbal interactions of individual participants.

Visual Text (written)

On-screen Captions

Depicts how much time (and space) is allocated to on-screen captions.

Textual Meaning

Cinematographic Resources

Composition (Phase, Shot)

Organisation of visual information into coherent units of meaning.

Interpersonal Meaning

Cinematographic Resources

Camera Distance (Long shot, Medium shot, Close shot)

Realizes interpersonal relations such as social distance.

Camera Movement (Stationary, Pan, Tilt, Zoom-in, Zoom-out)

Realizes interpersonal involvement by channelling the viewer’s gaze.

Vertical Viewing Perspective (High angle, Eye-level, Low angle)

Realizes interpersonal power relations through camera angle and perspective.

Systems for analysing intersemiotic relations in dynamic videos

Figure 6

Figure 6

On-screen captions

47The extra-diegetic voice-over narration [00:00:07 – 00:00:32], which immediately follows the introductory phase, but precedes any internal dialogue, in turn summarises, and hence resemiotises, the actions and events that are shown on screen, as well as those that are yet about to unfold in the video, thus adding another contextual layer. The voice-over narration, which is scripted and formal in terms of register, also resemiotises certain aspects of the WHO official’s dialogue, which uses a more informal lexis (see Fig. 7). For example, in the voice-over narration, participants are represented in discourse as impersonalised and collectivised (“WHO teams”, “health care workers”; Fig. 7 left). As such, they provide a contrast to the WHO official’s utterances, which are rendered more interpersonally engaging, partly through the use of first and second person plural pronouns (“we”, “you”; Fig. 7 centre). In this particular example, the process of intersemiotic translation is not unidirectional, or bidirectional, but multidirectional, as part of the WHO official’s dialog is again translated into a more formal, institutional register in the on-screen captions, which coincide with some of the WHO official’s utterances. In this case, the on-screen captions are all clauses in passive voice (e.g. see van Leeuwen 2008) which omit the agent or doer of an action altogether (Fig. 7 right). This has the effect that the experiential content is resemiotised in even more abstract terms than in the voice-over narration, as the “deagentialised” actions described in the on-screen captions are no longer identifiable with any particular participant(s).

48As illustrated in Fig. 7, these multiple resemiotisations across verbal and visual resources, across time and space, thus frame the WHO official’s informal, interpersonally engaging face-to-face dialogue within more authoritative, institutional registers, with a heightened focus on experiential meaning, resulting in a change in metafunctional orientation.

49The overall meaning of a video, however, is achieved through the complex co‑deployment of textual, experiential and interpersonal elements which are designed to make some meanings more prominent, and which guide the viewer to arrive at particular interpretations. In dynamic audio-visual texts, these interpretations are likely to result from choices made in the deployment of cinematographic resources and post-editing devices. Whilst these resources fulfil textual meaning in that they structure the unfolding of the video, they simultaneously realise interpersonal meanings such as Social Distance, Interpersonal Involvement,

Figure 7

Figure 7

Intersemiotic relations and translations across verbal and visual resources in a video

50Power Relations and Subjectivity or Point of View, as conveyed by Camera Distance (e.g. close shot, medium shot, long shot), Camera Angle (e.g. high angle, eye-level, low angle), Camera Movement (e.g. stationary, pan, tilt, zoom-in, zoom-out), Vertical Viewing Perspective (e.g. high angle, eye-level, low angle).

51As illustrated in Fig. 8 (which shows a snapshot of the analysis in the Multimodal Analysis Video GUI), the combination of multimodal semiotic resources, such as the dialogic space accorded to participants and on-screen captions, and the shifts in cinematographic choices that occur at the time the WHO official delivers her speech, all work together so that the viewer focuses on the WHO official and what she has to say at that moment, and the statements that are resemiotised in the form of on-screen captions. For example, as the WHO official begins to address the “gaps” in the hand-washing demonstration performed by the health care workers, the camera reduces the social distance between the WHO official and the viewer by zooming in and changing from a long shot to a medium shot. The placement of participants (dressed in dark colours) to the left and right of the WHO official (clad in a white top) additionally work to channel the viewer’s perspective to focus on the WHO official and by extension, the on-screen captions. While she delivers her speech, the camera remains stationary in a single Shot (with no editing cuts), and at eye-level, giving prominence to the points made in the on-screen captions. The overall meaning created would have been different if the camera had remained at long distance (as shown in the frame displayed at the top-left in Fig. 8), or if it had executed a pan or tilt which, together with high or low camera angles and changed depth perspectives, could have created a distancing or even disorientating effect. As illustrated here and discussed in the previous examples, intersemiotic translation can be modelled theoretically and analytically using the concept of metafunction, which reveals which the nature of the shifts of meaning which take place within and across multimodal texts.

Figure 8

Figure 8

Multi-semiotic combinations in a video

4.5. Resemiotisation in mathematics

52One of the most sophisticated and exact forms of intersemiotic translation occurs in mathematics which has specialised semiotic resources in the form of scientific language, mathematical images (e.g. graphs and diagrams), and mathematical symbolic notation to (re-)construct the material world (e.g. O’Halloran 2015a, 2015b). As O’Halloran explains:

[…] mathematics is considered to be a specialized tool for thinking, specifically designed to move beyond our everyday experience of the world to an abstract semiotic realm for restructuring thought and reality. Natural language also functions to organize and structure human experience on an abstract semiotic plane (e.g. Halliday, 1978; Whorf, 1956) but, unlike mathematics, it lacks the “meaning potential” (Halliday, 2003 [1985]) to effectively model and predict events in the physical world. (O’Halloran, 2015b, p. 288)

53Mathematics achieves its unique functionality by drawing upon the three semiotic resources which work closely together to bridge textual representations (i.e. language and mathematical symbolism) and visual representations (i.e. graphs and diagrams), with a specific focus on experiential and logical meanings. Each semiotic resource has its own functionality in this regard: language to contextualise the mathematics problem and results; images to provide a perceptual account of the mathematical relations where the parts are viewed in relation to the whole; and mathematical symbolism which mathematical relations are configured and rearranged to solve problems. Significantly, these three semiotic resources have systems of meaning which specifically designed to fulfil these functions, while seamlessly integrating so that it is possible to shift between each resource with ease. Moreover, mathematical symbolism has a specialised grammar for encoding meaning economically and unambiguously (as do mathematical graphs and diagrams) so that intersemiotic translations across language, image and the symbolism are precise and exact in nature. This is achieved through sophisticated compositional organisation of mathematical texts and backgrounding the interpersonal metafunction (i.e. through consistent semiotic choices within a narrow semantic field) so that the focus is experiential and logical meaning.

54The “semantic circuit” across language, image and symbolism in mathematics (O’Halloran 2005) is critical for achieving the semantic expansions that are required for rewriting the physical world in mathematical terms. The semantic circuit permits the meaning potential of three different resources to be accessed and, in addition, the intersemiotic translations of semiotic choices across text, image and symbolism create semantic expansions which extend beyond those possible for each resource. For example, the three graphs from the WHO website in Fig. 9a depict the number of cases of Ebola in Guinea, Liberia and Sierra Leone respectively. Upon mouse over, the numbers of cases on the “Patient database” and “Situation report” for each day are visible, as displayed on Fig. 9b. While the mathematical graphs are contextualised by the surrounding language, as displayed in Fig. 9a, the focus of the following discussion is the intersemiotic translation which takes place across mathematical relations (i.e. the reported number of cases each day) and the visual representations of this numerical data.

Figure 9a

Figure 9a

The Ebola crisis: mathematical graphs and text.

Figure 9b

Figure 9b

The Ebola crisis: mathematical graphs and text. (The situation in Guinea)

55The number of Ebola cases recorded in the “Patient database: and the “Situation report” are recorded on a daily basis. This results in a series of identifying mathematical relations between time (x) and number of cases (y): for example, on June 21 2015 there were 12 cases in both databases (i.e. Patient database and Situation report), as displayed in Fig. 9b. The data has the form of a mathematical relation where x (date) corresponds to y (number of cases): i.e. x ↔ y. This mathematical relation (x ↔ y) is translated visually into a point on the graph in Fig. 9b. In this case, there is intersemiotic translation between a mathematical relation x ↔ y (at the rank of clause) to an entity, the point (at the rank of visual entity), as shown in Table 3 which displays the systems for experiential meaning for language, image and mathematical symbolism. From here, the series of visual entities (i.e. the points representing the mathematical relation) form a visual relationship at the rank of episode, and the points are joined to form a graph at the rank of work, as displayed in Table 3.

56The example illustrates how intersemiotic translation involves shifts of meaning where semiotic choices at one rank for one semiotic system are resemiotised at a different rank in another semiotic system, with the result that the semiotic choice is re-contextualised within a new semantic field so that meaning potential of the new resource can be accessed, leading to further semiotic expansions. From there, other semiotic translations take place, as for example, when the complete work of the graph is semiotised as the word “graph”, in this case opening up the vast meaning potential of language to resemiotise the Ebola situation further, as illustrated in Fig. 9a. Such intersemiotic translations and resemiotisations form the basis for rewriting the material world in mathematical and scientific terms, consequently changing the nature of life on earth today.


Table 3

Table 3

Experiential meaning: language, image and symbolic systems in mathematics

5. Conclusion

58Intersemiotic translation is the basis of cultural communication through which thought and reality are structured using a variety of semiotic resources which function to construct our experience of the world and interpersonal relations through resemiotisation processes. As illustrated above, systemic functional theory (SFT) provides the necessary theoretical tools for modelling semiotic resources and analysing the meaning arising from semiotic choices within and across multimodal texts. In this case, semiotic resources are modelled as systems of meaning with various strands (i.e. experiential, logical, interpersonal and textual meanings) from which choices are made in multimodal texts and processes at different levels. In this way, it is possible to conceptualise shifts of meaning across semiotic resources which are fundamentally different in nature to reveal which meanings are retained and changed as a result of resemiotisation. That is, systemic functional theory provides comprehensive foundations for modelling intersemiotic translations theoretically and for demonstrating the semantic expansions that occur as a result, as illustrated in the examples discussed above.

59However, even with this multi-level theoretical platform, modelling and analysing intersemiotic translations within and across multimodal processes and texts and tracking the resultant effects are complex and time-consuming tasks. For this reason digital tools in the form of multimodal annotation software are employed, as discussed in the examples above. Even so, intensive close multimodal analysis is still labour-intensive and moreover it is difficult, and in some cases impossible, to track semantic patterns within and across multimodal texts. For this reason, mathematical modelling (e.g. clustering techniques and temporal logic) and visualisation techniques (e.g. state transition diagrams) have been used in order to capture semantic patterns in multimodal data which otherwise would not have been easily detected (e.g. O’Halloran, Marissa and Tan 2014; O’Halloran, Tan and Marissa 2015a; Tan, Smith & O’Halloran 2015). These modelling and visualisation techniques have only been applied to a limited number of multimodal texts, however. In this regard, the latest challenge is to combine close multimodal analysis with data mining and information visualisation to analyse large datasets of multimodal texts (e.g. O’Halloran, Tan, Pham, Bateman & Vande Moere, 2016). Interdisciplinary collaboration is required to develop new digital techniques and approaches to develop the science of intersemiotic translation for big data analytics. Such a science has implications in a range of fields where there is a need to understand patterns of human discourse (e.g. national security and defense, terrorism studies, disaster management, healthcare) in a world where communication has been radically transformed through advances in digital technology. In this respect, semioticians have a major role to play with regards to modeling, tracking and understanding intersemiotic translation as the basis for cultural communication.

Top of page



Aguiar, Daniella & Queiroz, Joao (2009), “Towards a Model of Intersemiotic Translation”, The International Journal of the Arts in Society, 4(4), pp. 203-210.

Bateman, John & Schmidt, Karl-Heinrich (2012), Multimodal Film Analysis: How Films Mean, London and New York: Routledge.

Borodo, Michał (2015), “Multimodality, Translation and Comics”, Perspectives, 23:(1), pp. 22-41, DOI: 10.1080/0907676X.2013.876057.

Eco, Umberto (1979), A Theory of Semiotics, Bloomington: Indiana University Press.

Eggins, Suzanne (2004), An Introduction to Systemic Functional Linguistics, 2nd edition, London and New York: Routledge.

Fine, Elizabeth (1984), The Folklore Text, Bloomington: Indiana University Press.

Gottlieb, Henrik (1997), Subtitles, Translation and Idioms, Copenhagen: University of Copenhagen.

Halliday, Michael A.K. (1978), Language as Social Semiotic: The social Interpretation of Language and Meaning, London: Edward Arnold.

Halliday, Michael A.K. (2002 [1977]), “Text as Semantic Choice in Social Contexts”, in Webster (ed.), Linguistic Studies of Text and Discourse: Volume 2 in the Collected Works of M.A.K. Halliday, London and New York: Continuum, pp. 23-81.

Halliday, Michael A.K. (2003 [1985]), “Systemic Background”, in Webster (ed.), On language and Linguistics: the Collected Works of M.A.K. Halliday volume 3, London and New York: Continuum, pp. 185-198.

Halliday, Michael A.K. (2008), Complementarities in Language, Beijing: Commercial Press.

Halliday, Michael A.K. & Hasan, Ruqaiya (1985), Language, Context, and Text: Aspects of Language in a Social-semiotic Perspective, Geelong, Victoria: Deakin University Press [Republished by Oxford University Press 1989].

Halliday, Michael A.K. & Matthiessen, Christian M.I.M. (2014), Halliday’s Introduction to Functional Grammar, 4th ed., revised by Matthiessen (ed.), London and New York: Routledge.

Iedema, Rick (2003), “Multimodality, Resemiotisation: Extending the Analysis of Discourse as Multi-semiotic Practice”, Visual Communication, pp. 29-57. DOI: 10.1177/1470357203002001751.

Jakobson, Roman (1959), “On Linguistic Aspects of Translation”, in Brower, (ed.), On Translation, Cambridge, MA: Harvard University Press, pp. 232-239.

Jewitt, Carey (2014), “An Introduction to Multimodality”, in Jewitt (ed.), The Routledge Handbook of Multimodal Analysis, 2nd ed., London: Routledge, pp. 15-30.

Jewitt, Carey (ed., 2014), The Routledge Handbook of Multimodal Analysis, 2nd ed., London and New York: Routledge.

Jewitt, Carey, Bezemer, Jeff & O’Halloran, Kay L. (2016), Introducing Multimodality, London: Routledge.

Kourdis, Evangelos (2015), “Semiotics of Translation: An Interdisciplinary Approach to Translation”, in Trifonas (ed.), International Handbook of Semiotics, Dordrecht: Springer, pp. 303-320.

Kourdis, Evangelos & Yoka, Charikleia (2014), “Intericonicity as Intersemiotic Translation in a Globalized Culture”, in Wang & Ji (eds.), Our World: A Kaleidoscopic Semiotic Network, Proceedings of the 11th World Congress of the IASS/AIS, 5-9 October 2012, Hohai University Press, pp. 162-176.

Kress, Gunther (2009), “What is a Mode?”, in Jewitt (ed.), The Routledge Handbook of Multimodal Analysis , London: Routledge, pp. 54-67.

Kress, Gunther & van Leeuwen, Theo (2001), Multimodal Discourse: The Modes and Media of Contemporary Communication, London: Edward Arnold.

Kress, Gunther & van Leeuwen, Theo(2006), Reading Images: The Grammar of Visual Design, 2nd ed., London: Routledge.

Lemke, Jay L. (1998), “Multiplying Meaning: Visual and Verbal Semiotics in Scientific Text, in Martin & Veel (eds.), Reading Science: Critical and Functional Perspectives on Discourses of Science, London: Routledge, pp. 87-113.

Martin, James (1992), English Text: System and Structure, Amsterdam: Benjamins.

Martin, James (2002), “Meaning beyond the Clause: SFL Perspectives”, Annual Review of Applied Linguistics, 22, pp. 52-74.

Martin, James R. & Rose, David (2007), Working with Discourse: Meaning beyond the Clause, 2nd ed., London: Continuum.

Martin, James R. & White, Peter R.R. (2005), The Language of Evaluation, Appraisal in English, London and New York: Palgrave Macmillan.

Martinec, Radan (2001), “Interpersonal Resources in Action”, Semiotica, 135(1/4), pp. 117-145.

Martinec, Radan(2004), “Gestures that Co-concur with Speech as a Systematic Resource: The Realization of Experiential Meanings in Indexes”, Social Semiotics, 14(2), pp. 193-213.

Matthiessen, Christian M.I.M. (2009), “Multisemiotic and Context-based Register Typology: Registerial Variation in the Complementarity of Semiotic Systems”, in Ventola & Moya (eds.), The World Told and the World Shown: Multisemiotic Issues, Hampshire: Palgrave Macmillan, pp. 11-38.

O’Halloran, Kay L. (2005), Mathematical Discourse: Language, Symbolism and Visual Images, London and New York: Continuum.

O’Halloran, Kay L. (2015a), “The language of learning mathematics: a multimodal perspective”, The Journal of Mathematical Behaviour, 40 Part A, pp. 63-74.

O’Halloran, Kay L. (2015b), “Mathematics as Multimodal Semiosis”, in Davis and Davis (eds.), Mathematics, Substance, and Surmise, Berlin: Springer, pp. 287-303.

O’Halloran, Kay L., & Lim-Fei, Victor (2014), “Systemic Functional Multimodal Discourse Analysis”, in Norris & Maier (eds.), Texts, Images and Interactions: A Reader in Multimodality, Berlin: Mouton de Gruyter, pp. 137-154.

O’Halloran, Kay L., Marissa K.L.E. & Tan, Sabine (2014), “Multimodal Analytics: Software and Visualization Techniques for Analyzing and Interpreting Multimodal Data”, in C. Jewitt (ed.), The Routledge Handbook of Multimodal analysis (2nd edition), London: Routledge, pp. 386-396.

O’Halloran, Kay L., Tan, Sabine & Marissa K.L.E. (2015a), “Multimodal Semiosis and Semiotics”, in Webster (ed.), The Bloomsbury Companion to M.A.K. Halliday, London: Bloomsbury, pp. 386-411.

O’Halloran, Kay L., Tan, Sabine & Marissa K.L.E. (2015b), “Multimodal Analysis for Critical Thinking”, Learning, Media and Technology, DOI: 10.1080/17439884.2016.1101003.

O’Halloran, Kay L., Tan, Sabine, Pham, Duc-Son, Bateman, John & Vande Moere, Andrew (2016), “A digital Mixed Methods Research Design: Integrating Multimodal Analysis with Data Mining and information visualization for big Data Analytics”, Journal of Mixed Methods Research. DOI: 10.1177/1558689816651015.

O’Toole, Michael (2011), The Language of Displayed Art, 2nd ed., London and New York: Routledge.

O’Sullivan, Carol (2013), “Introduction: Multimodality as Challenge and Resource for Translation”, The Journal of Specialised Translation, 20, pp. 2-14.

Perez-Gonzalez, Luis (2014), “Multimodality in Translation and Interpreting Studies”, in Bermann & Porter (eds.), A Companion to Translation Studies, Chichester: Wiley-Blackwell, pp. 119-131.

Remael, Aline (2001), “Some Thoughts on the Study of Multimodal and Multimedia Translation”, in Gambier & Gottlieb (eds.), (Multi) Media Translation: Concepts, Practices and Research, Amsterdam: John Benjamins, pp. 13-22.

Royce, Terry (2007), “Intersemiotic Complementarity: A Framework for Multimodal Discourse Analysis”, in Royce & Bowcher (eds.), New Directions in the Analysis of Multimodal Discourse, T. Mahwah NJ: Lawrence Erlbaum Associates, pp. 63–110.

Tan, Sabine, Smith Bradley, A. & O’Halloran, Kay L. (2015), “Online Leadership Discourse in Higher Education: A Digital Multimodal Discourse Perspective”, Discourse & Communication, 9(5), pp. 559-584, DOI: 10.1177/1750481315600302.

Taylor, Christopher J. (2003), “Multimodal Transcription in the Analysis, Translation and Subtitling of Italian Films”, The Translator, 9(2), pp. 191-205, DOI: 10.1080/13556509.2003.10799153.

Taylor, Christopher J. (2013), “Multimodality and Audiovisual Translation”, in Gambier & van Doorsaer (eds.), Handbook of Translation Studies, Volume 4, John Benjamins, pp. 98-104.

van Leeuwen, Theo (1999), Speech, Music, Sound, London: Macmillan.

van Leeuwen, Theo (2008), Discourse and Practice: New Tools for Critical Discourse Analysis, New York: Oxford University Press.

Whorf, Benjamin L. (1956), Language, Thought and Reality: Selected Writings of Benjamin Lee Whorf, MA: MIT Press.

Top of page

List of illustrations

Title Figure 1a
Caption News story ‘Ebola – a test too far for one little girl’4 as displayed on the WHO Ebola website
File image/jpeg, 196k
Title Figure 1b
Caption Intersemiotic translation between photograph and text as analysed in Multimodal Analysis Image software
File image/jpeg, 136k
Title Figure 2
Caption Resemiotisation in an infographic
File image/jpeg, 76k
Title Figure 3
Caption Ebola symptoms resemiotised as images
File image/jpeg, 56k
Title Figure 4
Caption Resemiotisation between different media – graphic and photograph
File image/jpeg, 40k
Title Figure 5
Caption Resemiotisation between different media – graphic and video
File image/jpeg, 44k
Title Figure 6
Caption On-screen captions
File image/jpeg, 332k
Title Figure 7
Caption Intersemiotic relations and translations across verbal and visual resources in a video
File image/jpeg, 180k
Title Figure 8
Caption Multi-semiotic combinations in a video
File image/jpeg, 2.6M
Title Figure 9a
Caption The Ebola crisis: mathematical graphs and text.
File image/jpeg, 72k
Title Figure 9b
Caption The Ebola crisis: mathematical graphs and text. (The situation in Guinea)
File image/jpeg, 36k
Title Table 3
Caption Experiential meaning: language, image and symbolic systems in mathematics
File image/jpeg, 126k
Top of page


Bibliographical reference

Kay L. O’Halloran, Sabine Tan and Peter Wignell, Intersemiotic Translation as Resemiotisation: A Multimodal PerspectiveSignata, 7 | 2016, 199-229.

Electronic reference

Kay L. O’Halloran, Sabine Tan and Peter Wignell, Intersemiotic Translation as Resemiotisation: A Multimodal PerspectiveSignata [Online], 7 | 2016, Online since 31 December 2017, connection on 04 December 2022. URL:; DOI:

Top of page

About the authors

Kay L. O’Halloran

Kay O’Halloran is Professor in the School of Education, Faculty of Humanities at Curtin University. Her areas of research include multimodal analysis, social semiotics, mathematics discourse, and the development of interactive digital media technologies and visualization techniques for multimodal and sociocultural analytics.

Sabine Tan

Sabine Tan is a Research Fellow in the School of Education, Faculty of Humanities at Curtin University. Her research interests include critical multimodal discourse analysis, social semiotics, and visual communication. She is particularly interested in the application of multidisciplinary perspectives within social semiotic theory to the analysis of institutional discourses involving traditional and new media.

Peter Wignell

Peter Wignell is a Research Fellow in the School of Education, Faculty of Humanities at Curtin University. Peter’s current research interests are in Systemic Functional Linguistics, especially in its application to the analysis of multimodal texts. His research has also focused on the role of language in the construction of specialised knowledge.

Top of page



Creative Commons - Attribution 4.0 International - CC BY 4.0

Top of page
Search OpenEdition Search

You will be redirected to OpenEdition Search