1The avatar is at the heart of Georges Gagneré and Cédric Plessiet’s research and artistic productions. These two artists and researchers are on the Digital Arts and Technologies teaching team, and are members of the Digital Image and Virtual Reality team within the Visual and Contemporary Arts research lab at Paris 8 University. Georges Gagneré has been exploring the role of avatars as acting partners opposite physical performers on a mixed-reality stage that brings together real and virtual worlds. Cédric Plessiet, on the other hand, focuses on the realistic digital double existence of actors—or of physical models, in the context of interactive installations. Their first collaboration, which dates back to 2015 (Gagneré & Plessiet, 2015), resulted in a fruitful cross-fertilization of their explorations into numerous research questions, such as: how can avatars be used in a way that lets them exist artistically and that brings forward new forms of dramaturgy, promoting interactions with the audience? How can we draw on real-time digital tools from animated movies and video games when designing these avatars?
2By simultaneously positioning our research in the fields of animated film, video games, visual arts, and performing arts, we observed a proliferation of definitions and uses of the avatar. This term also appears as a widely used concept in a growing number of scientific fields, including computer science disciplines dealing with language, conversational agents, cognition, and visual rendering. As a result, we must first outline our definition and use of this protean concept based on the specific computer tools used in our research-creation projects. Recent developments have led to a paradigm shift concerning these software tools, which can be linked to the emergence of increasingly high-quality methods of real-time rendering. These developments are central to our artistic practice, which aims to enable interactions between virtual avatars and the real world.
- 1 Les Double-Autres, Microfolie Festival 2018 (Villiers-le-Bel), Exhibition Play!, 2019 (Bar-le-Duc), (...)
- 2 L’Ombre, based on the work of H. C. Andersen. Directing, scenography, and digital programming: Geor (...)
3Consequently, we will first introduce the challenges raised by real-time technology in the context of on-set pre-visualization, a defining aspect of our work, before going on to describe our avatars and their behaviors. Secondly, we have chosen to present two separate but parallel research-creation projects: Les Double-Autres [The Double Others],1 a collection of interactive portraits designed by Cédric Plessiet, and L’Ombre [The Shadow],2 a performance directed by Georges Gagneré. Lastly, the description of a new joint research-creation project will demonstrate the convergence of our investigations.
4On-set pre-visualization is an emergent response to the needs of the film industry and the video game industry, as a way to improve the technological and artistic integration of digital technology. As a tool for the production of audiovisual content, it provides movie directors and producers with an immediate rendering of images that will come close to the end result intended for an audience. This means that the difference between so-called pre-rendered productions (typically movies, which are made up of image sequences in which each sequence requires extensive computation time) and “real-time” productions (typically video games, because each image is the result of near-instant calculation) tends to fade. The film industry in particular has been affected by the new architecture of many software processes and tools designed to meet production needs. These “pipelines” make it possible to capture and arrange images in real time (Antoniades, 2016) thanks to hyper-realistic digital doubles of existing human individuals (Seymour, 2017). One of the aims of pre-visualization technology is to help physical performers understand the virtual environments (including special effects and virtual creatures) inhabited by their virtual characters (Bouville, 2016).
5The tools and methods of on-set pre-visualization can be used for any kind of artistic practice that calls for physical/digital hybridization processes, during the stages of design, production, and exhibition, and can also appeal to disciplines that lie outside the fields of film and video games, such as live performance, visual arts, or music. Our collaborative projects make active use of on-set pre-visualization, even at the final stage when presenting the work to the public. The tools we use are not only stepping stones used to approximate the end result: they are an integral part of the production presented to the audience. We will nevertheless continue to use the term “pre-visualization” in the context of our research-creation projects because it encompasses the real-time dimension of the processes implemented as well as their existence on a timeline that extends from the initial stage of design to that of production, all the way to finalization. Furthermore, these artistic productions systematically involve avatars.
6What do we mean by the word avatar? The term shows up in numerous contexts and is often used as a synonym for virtual actor. Among these myriad definitions, we are particularly interested in the term as it first appeared in the Lucasfilm game Habitat, where it refers to a user’s interface or vehicle in the virtual world of a video game. In their pioneering paper on the emergence of new recreational behaviors (Perény & Amato, 2010), Étienne Armand Amato and Étienne Perény also provide an in-depth examination of this concept, insisting that:
Increasingly, the real, as well as the potential, will be split in two by the virtual, in the English sense of the word: that of another reality, which is simply a quasi-reality, or the reality of a virtual and operable image, operated and inter-operated [agie et inter-agie], and ultimately liable of turning of anyone and anything into an avatar that can manifest itself on a screen. (p. 112)
7An avatar is also used to describe the virtual entity that lets us visualize a process of motion capture applied to a human individual, bringing us back to the idea of transferring motion-based data from a physical body onto a digital one.
8Our avatars are informed by the notion of matrix-sculpture (Plessiet, 2019). Based on digital representations that allow users to create a generally anthropomorphic entity, the matrix-sculpture consists of a mesh of points, or polygons, making up a virtual outer envelope. Coupled with joints that collectively make up a rig, this polygon envelope can assume an infinite variety of positions in space. It provides the matrix for the corporeal and facial movements that result from the way the rig, or skeleton, is controlled across time.
9The avatar as a concept assumes various names in various fields. In the 1980s, Daniel Thalmann and Nadia Magnenat-Thalmann’s groundbreaking work led to some of the earliest uses of realistic avatars in 3D animated films (Magnenat-Thalmann, 1987), and suggested classifying virtual actors into four main categories: participant actors, controlled through motion-capture technology; guided actors, controlled through peripheral devices; autonomous actors, capable of interacting with their environment; and lastly, actors capable of interacting with other real or virtual actors. The aim of this computational approach was to create “intelligent” virtual human beings capable of simulating real human individuals (Couchot, 2013). Researchers in computer and cognitive sciences later shifted their focus to animated conversational agents. Instead of simulating human beings in full, the idea was to develop complex interactions using only their faces. Research conducted by Catherine Pelachaud (1996) in this field led to the development of GRETA (Rosis, 2003), an expressive and interactive conversational agent platform capable of communicating with users both verbally and non-verbally, by way of looks, movements of the head and chest, facial expressions, and gestures. Artificial intelligence also addresses the matter of complex interactions between multiple agents by attempting to take into account the enactive paradigm of self-organization in an environment (Varela, 1993). One example of this is the model MASCARET, developed by Pierre Chevaillier and Pierre de Loor, which is dedicated to educational experimentation (Buche, 2004).
10Other researchers have focused on ways of describing avatars’ actions by drawing on theatrical stage techniques (Ronfard, 2016), leading them to examine how emotions can be caused by neutral expressions, working to direct the facial expressions of a virtual performer in the manner of a stage director (Barbulescu, 2014). In the field of live performance, the growing presence of virtual characters has challenged our concept of what defines actors and their onstage presence (Ferral, 2012). More broadly, the need to clarify the role of virtual performers in digital art practices such as animated film and video games has led us to develop a classification system based on two criteria: origin of movement and origin of decision (Plessiet, 2019). According to whether the origin of the choice or of the movement itself is internal or external to the entity, four virtual figures stand out: the puppet, whose movements and decisions have an external source; the autonomous actor, characterized in both cases by internal causality; the golem, whose movements are self-produced but who is bereft of free will; and the mask, whose movements originate from outside itself, but whose decisions are its own. As an example, characters in the video game The Sims (Wright, 2000) can be seen as golems obeying the player; objects, on the other hand, contain action-oriented decisions that require the characters’ external movements to actualize themselves (fig. 1). These objects are virtual masks.
Figure 1
The “Puppet, Golem, Actor, Mask” model (PGAM).
Graphics credit: Cédric Plessiet.
11To better understand how the figure of the avatar can play a central part in artistic processes, the next part of this paper describes our use of the matrix-sculpture in two research-creation projects: Les Double-Autres, a collection of interactive portraits by Cédric Plessiet, and L’Ombre, a performance directed by Georges Gagneré (Gagneré, 2020) based on Hans Christian Andersen’s homonymous tale. The first project explores the animation of realistic virtual actors as part of an interactive installation. The second project features non-realistic avatars engaged in a hybrid relationship with an actor over a 50-minute theatrical performance. In both cases, the use of matrix-sculptures and the presentation of the artwork to the public make generalized use of on-set pre-visualization technology.
12The “Double-Other” is a virtual entity based on an actual person, the model, and which forms the basis for a gallery of interactive portraits. The avatar’s design involves a hybrid approach combining 3D digitization, artistic process, and task automation.
13First, I shall retrace the main steps by which I am able to achieve a photorealistic digital double of the physical model that my three-dimensional interactive portrait is based on. Traditional processes borrowed from the video game industry suggest several ways of creating a mesh using 3D capture systems, such as photogrammetry or a 3D scanner. My approach also separates body design from face design. I design the body using preexisting meshes that I customize with Blender and Zbrush. For the face mesh, I use a 3D scanner, obtaining a mesh that is indeed lifelike but unfortunately inoperable in its present state because it is not designed to deform. The geometry of the 3D scanner does not contain the curvature lines that allow for the type of distortions required by facial expressions. I must therefore edit its topology, so that it can later be stretched out, crushed, or curved.
14The next step consists in coupling this customized mesh with a rig that allows for the deformation and animation of the matrix-sculpture. The animation controls take into account the type of freedom of movement I want to work into the specific anatomy of the mesh derived from the physical model. To ensure consistent facial movements, I break down possible expressions according to the Facial Action Coding System or FACS (Eckman, 2002). This system relies on a limited number of facial action units (FAU) that make up so many elements of expression related to the face’s anatomy. This means that I must conduct a scan of each of the thirty or so expressions specific to the physical model which these FAUs are based on.
15With preexisting mesh elements related to the body and a score of new face elements, I must now design an avatar that should be both a highly resemblant double of the physical model and one sufficiently manipulable to allow it to perform its own expressive movements, turning it into what I call a “Double-Other.” The Double-Other is a matrix-sculpture which couples a great faithfulness to the physical model with the possibility of performing complex movements that fall in line with the different possibilities set out by the PGAM model (see section “Avatar Classifications”). The virtual Double-Other splits the physical model in two, and emancipates itself through movements of its own.
16Wishing to focus on more interesting creative tasks and to obtain rapid results, I next considered how to optimize the creation of a Double-Other. This need for rapid reaction times is at the heart of on-set previsualization and makes it possible to monitor changes in the end result at each stage. It ties in with the agile philosophy (Autissier, 2021), according to which the capacity to iterate quickly on a working product allows both for creative dynamism and artistic control. These considerations led to the development of Taupipeline, which brings together various processes (modeling, rigging, animation, rendering) required to create a Double-Other.
17Taupipeline (fig. 2) crucially relies on the free software tools Tactic and Afanasy, used for asset management and rendering distribution, respectively. I use standard computer graphics software, such as Maya, Blender, and Photoshop, to produce 3D components. Practically speaking, I start by searching through existing assets on Taupipeline for components that might fit with the new physical model. If I cannot find any matches, I recreate or edit new components which I then add to the database. For the face, I need to design the thirty or so FAUs required for the facial expressions. I use a process automation software that I programmed which saved me several hours of repetitive, non-creative, and tedious work. Automation also makes it easier to avoid manipulation errors. Finally, the Tauporchestre module carries out this set of automated tasks on several computers at once. Any change brought to a specific element is thus easily and rapidly computed and applied to each and every one of the matrix-sculpture’s components.
Figure 2
Taupipeline and Taupemodel modules.
Graphics credit: Cédric Plessiet.
18Taupipeline represents a compromise between 3D character design solutions like Metahuman (Metahuman, 2021), which provide immediate but stereotypical results, and the ex nihilo design of a 3D character with a more personal feel, which takes a very long time and requires an elaborate skillset.
19Animations and interactions are then finalized thanks to the virtual-reality and motion-capture processes brought together in the Taupemodel module (fig. 2), which consists of a motion-capture system and the real-time game engine Unreal Engine. This engine was chosen because of its graphic capabilities, especially for skinning, and because of the easy access to its source code, which allows for advanced modifications. Moreover, the native integration of third-party interface software makes it easier to associate with my own technology.
20Equipped with a motion-capture device connected to the video monitor and a virtual reality headset (fig. 3a), I strike a pose or perform whatever motion I wish to pass on to the Double-Other. I can then study it from an external point of view, and manipulate the rig to correct potential animation issues (a collision between body and limb, the position of the fingers, etc.). I then adjust the different components of the scene, the lighting, and the camera frame. All of these operations are done in virtual reality, which allows me to have a clear and tangible understanding of volumes and of the 3D render. By living in the skin of the Double-Other I wish to animate, I enter into a relationship of empathy, even though the avatar is for the most part nothing like me (fig. 3b).
Figure 3a
Figure 3b
(a) Cédric Plessiet in Taupemodel; (b) Eden Weiss’s Double-Other, controlled in real time by Cédric Plessiet in Taupemodel.
Graphics credit: Cédric Plessiet.
21The interactive portrait Ça bug dans les rotations [A Bug in the Rotations] features one of Georges Gagneré’s Double-Others (fig. 4, left), his head subjected to untimely rotations which the viewer can attempt to rectify (or intensify) by operating a rotary switch. The portrait harks back to conversations raised by rotation issues during a previous collaboration, in relation to a virtual puppet. La Geôle [The Jail] explores the autonomy of my own Double-Other (fig. 4, right). This self-portrait in-the-making shows me writing a book. I first started working on it while I was in the process of writing my dissertation. My Double-Other develops his own writing routine through a process of trial-and-error in an absurd universe. The learning pattern relies, among other things, on the use of artificial neural networks.
Figure 4
Left, Georges Gagneré’s Double-Other in Ça bug dans les rotations; right, Cédric Plessiet’s Double-Other in La Geôle, with visible decision-making operations.
Graphics credit: Cédric Plessiet.
22This accelerated avatar production process allows for rapid and inexpensive changes. The increased speed and flexibility make it possible to bring changes to the design method, for instance by inviting the original human model back into the process, letting her engage with me, give her opinion, request adjustments, or explore her virtual body. Some models have wanted to change their hairstyles, others have wanted to see themselves pregnant, etc. The dialogue between the model and the artist is made possible by Taupipeline. The Double-Other becomes a co-creation in which the virtual body balances my vision of the model with her own self-perception and the way she wishes to avatarize herself.
23I will now describe my use of avatars in the research-creation project L’Ombre (Gagneré, 2020), starting with the aesthetic framework of the performance and the avatar control system. I will then examine the artistic questions raised by matters of stage direction, as well as the solutions developed to animate the avatars and work the animations into the actor’s performance.
- 3 The French word castelet refers to a component of puppet theater scenery which creates a frame arou (...)
24The show uses the concept of CAstelet Virtuel d’OmbrAVatars (Virtual Puppet Booth for Shadow-Avatars, CAVOAV) (Gagneré & Ternova, 2020) which borrows from traditional shadow play theater and the dramaturgical possibilities afforded by the figure of the shadow, drawing particular inspiration from Ernst Theodor Amadeus Hoffmann (1776-1822), Adelbert von Chamisso (1781-1838), and Hans Christian Andersen (1805-1875). This approach lets me breathe three-dimensional life into shadow-like avatars, dubbed ombravatars [shadow-avatars] (fig. 5). Shadow-avatars are matrix-sculptures whose mesh consists of a two-dimensional human silhouette replicating the exact shadow cast by an individual when he or she is lit up by a spotlight positioned perpendicular to a screen. The mesh is fitted with a rig, which has the specificity of belonging to a plane. This “flat” rig nevertheless allows for a three-dimensional manipulation of the silhouette thanks to motion-capture devices that make it possible to transpose the boundless range of a person’s natural movements. CAVOAV is a three-dimensional space encased in a virtual frame. Projected onto a stage, the space opens up a kind of theater within a theater, populated by shadow-avatars that can engage with the physical actors performing in front of the screen. Thanks to the real-time engine’s 3D rendering, the visual effect conjures up the existence of a virtual puppet booth3 set up behind the rigid virtual frame of the image.
Figure 5
Excerpt from the performance L’Ombre, directed and programmed by Georges Gagneré.
Graphics credit: didascalie.net.
25A shadow-avatar is an anthropomorphic entity capable of existing both as a shadow, projected in two dimensions, and as a three-dimensional silhouette, as shown in Figure 5: a shadow-avatar (in gray) is shown acting opposite its own shadow cast against the windows (in black). Like any virtual entity, shadow-avatars can lend themselves to any special effect imaginable and programmable. The animation of the silhouettes is based on AvatarStaging4 (Gagneré, 2018), a system which combines several software tools in order to transpose movements performed by an actor equipped with a motion-capture device—the “mocaptor”—onto the avatars inside the 3D digital space CAVOAV, which represents the virtual component of the mixed-reality stage.
Figure 6
AvatarStaging.
Graphics credit: Georges Gagneré.
26Generically speaking, AvatarStaging brings together three processes, successively detailed in each of the next three paragraphs. They let a mocaptor control an avatar and develop a relationship between said avatar and a physical actor on a mixed-reality stage (A + B). Figure 6 is a schematic representation of how these three processes operate for a mocaptor, an avatar, and an actor, variations of which can be inferred in more complex configurations. In the first process, the mocaptor is wearing a full-body suit, which can be optical or inertial, and which makes it possible to collect data relating to the position of the sensors situated on specific areas of the mocaptor’s morphology (fig. 6, Zone C). The motion-capture data is then transferred to a first specific matrix-sculpture, i.e., a virtual skeleton inferred from the position of the sensors situated on the mocaptor’s outer envelope. This skeleton is hence the result of a mathematical approximation and does not match the mocaptor’s actual posture or frame. Moreover, multiple parasitic factors affecting data capture can cause distortions. Their impact on the artistic quality of the movements must constantly be assessed and corrected, if necessary.
27The second process, called motion retargeting, involves transposing the movements performed by the mocaptor’s first virtual skeleton onto the virtual skeleton of the matrix-sculpture corresponding to the avatar chosen to perform in the 3D space opposite the actor. In L’Ombre, the shadow-avatar has a very specific build, inevitably distorting the mocaptor’s original motions, an aspect which I will discuss at greater length in relation to the project’s artistic outcomes (see section “Methodology of Animation Design”).
28The third and last process consists in organizing the positions of the avatar and/or its prerecorded animations according to the digital setting and to the positions of the physical actor (fig. 6, zone B), using the AKN_Regie plugin (which I programmed in a game engine).
29To sum up: the mocaptor present in space C develops a stage relationship to the actor on stage A thanks to the video playback monitors placed around him, which enable him to see the result of his movements projected onto his avatar in digital space B—a dynamic made possible by the three processes of motion capture, motion retargeting, and motion control with AKN_Regie.
30The dramatic hypothesis underlying L’Ombre goes as follows: a physical actor narrates Andersen’s homonymous tale to a physical audience and to shadow-avatars inside CAVOAV. Over the course of the narration, the shadow-avatars begin to reenact a series of scenes, taking on the part of five protagonists from this fantastic tale about a scholar from the North who leaves his shadow behind in some torrid land of the South. Certain parts of the story contain dialogue, so that the narrator performs the characters’ voices in turn; their movements, however, originate from other sources. One might picture a group of five mocaptors controlling the five shadow-avatars from behind the scenes with the use of AvatarStaging, while a physical narrator is narrating the story. But because I wished to achieve a light-weight form which could be performed in any place equipped with a projection surface, I wanted the actor to be able to man the technical aspects on his own, in full view of the audience (visual animations, music score, light and sound effects). In other words, the dramatic hypothesis meant that the actor had to be able to activate the prerecorded animations that simulate attentiveness and active participation from the shadow-avatars. This led to a first problem: reconciling the actor’s freedom of interpretation—in terms of pacing and acting intentions—with prerecorded animations, which imply firmly set intentions and a predetermined pace. A second challenge was to record lifelike animations capable of endowing the shadow-avatars with some kind of stage presence. We shall first examine how to achieve motion captures capable of breathing life into the shadow-avatars acting out the narrator’s story. Failing to achieve lifelike animations would have ruled out the possibility of combining them with the narrator’s live performance in a consistent way. These mechanisms will be briefly discussed further on.
31In film and audiovisual media, motion-capture animation usually involves putting the mocaptor in the same physical situation as the one their avatar will be performing in, before carrying out the three operations depicted above to design the corresponding animation in post-production. This approach involves a separation between the production of the movements on the one hand, and the reception of the animated avatar’s expressiveness by fellow performers on the other. This means that animations cannot be modified to take into account the kind of mutual influence that informs human interactions.
32Picture now our mocaptor stepping into the part of a shadow-avatar on the mixed-reality stage. The narrator is performing opposite another actor dressed in a motion-capture suit, in no way his final acting partner. A fundamental aspect of my approach involves making use of on-set previsualization to address the aesthetic nature of the shadow-avatar. More important than the mocaptor’s motions per se are these movements’ effects on the shadow-avatar, the narrator’s final acting partner. Over the course of the rehearsals and of the recording of the animations, AvatarStaging makes it possible for the physical actor to step into his role (fig. 6) and to perform on the final mixed-reality stage opposite the shadow-avatar. Conversely, the mocaptor uses video monitors to watch the results of the interactions and to adjust them in real time. In a traditional theater production, the possibility of working in final performance conditions often only comes up at the very end, a few days before dress rehearsal, when the costumes, set, lights, and special effects are finally brought together and adjusted. My approach consists in bringing all the components together from the start and making whatever changes are needed on the spot, over the course of the rehearsals.
33Next, the narrator and the shadow-avatar must develop their stage relationship under the director’s watch. Indeed, the director is also in charge of writing the dramatic hypothesis which determines how the shadow-avatars will listen to the story. In this context, a side note about how the theater traditionally addresses character-building seems necessary. Generally speaking, the challenge for any actor is to be able to control their emotions in order to develop them according to a dramatic intent co-elaborated with the stage director and acting partners, based on the material provided by the playwright. Actors must paradoxically balance control and honesty to develop their characters. In other words, they must simultaneously inhabit their own body and distance themselves from their emotions, in order to be able to arrange them in a way that brings out a character for the audience. In L’Ombre, the narrator’s acting technique falls in line with this theatrical tradition. But the mocaptor’s character inhabits a body that isn’t his own, hence disrupting the traditional relationship. Up to now, the relation to the avatar character has received little attention from theater studies (Gagneré, 2015; Kennedy, 2021). I will examine some of its aspects as explored in collaboration with the mocaptor who carried out the recordings: an experienced drama student who had already taken part in several motion-capture projects involving an inertial suit.
34Beyond the specifics of character development, the entities in relation are of different natures. Human beings do not naturally coexist with shadow-avatars. The stage director must assess the nature of this relationship in order to develop a convincing dramaturgy. Because of this, I chose to add a silent introduction to the performance: I suggested that the physical narrator touch the shadow-avatars with his own shadow cast by the light of the digital projector. The idea was to conjure up a situation where roaming shadows are progressively tamed and invited to share a moment with the audience. What I like about the nature of the shadow is precisely its ambivalent relation to human corporeality. The shadow is an extension of our presence in the world which also prompts us to reflect on its vulnerability. Are we alone with ourselves? Who is this other self, copying my motions to perfection? What are its powers? The shift from 2D to 3D afforded by the shadow-avatar is a way of moving the viewer to accept the fact that the narrator rubs shoulders with shadows who transform into bizarre creatures. The emergence of triangular stage relationships between the actor, the mocaptor, and the director requires iterative adjustments over the course of the rehearsals. This allows all the partners involved to improve their technique in real time, in a situation identical to the one experienced by the audience.
35Let us now turn to the mocaptor/shadow-avatar’s specific acting technique. The aim of the first process is to acquire command of the motion-capture device. This calls for constant attention and evaluation, as multiple parameters can affect the virtual skeleton obtained through motion capture. Furthermore, because of the inertial suit used in L’Ombre, the sensors’ sensitivity to magnetic fields induced small but recurrent distortions which called for continued vigilance on the mocaptor’s part. Next, the specific constraints of the motion-retargeting process must be taken into account: what will the relevant expressive movements turn out to be after their transposition to the specific morphology of the shadow-avatar? Among the many adjustments required to balance possible distortions or confusions, we found: bending the arms to bring life to the silhouette’s forced dimensionality; positioning the arms alongside the body to avoid collisions with the hands; emphasizing signs of global body orientation, such as an arm stretched forward, to make it clear where the shadow-avatar is looking and make up for ambiguity caused by the absence of depth; as well as designing the avatar’s walk to identify emotions by its way of moving through space. Developing a proprioceptive sense of the shadow-avatar’s envelope, that is to say a satisfying integration of the body’s perceptions, is a challenging task for the mocaptor. The apparent solution which consists in wrapping the mocaptor inside this envelope thanks to a virtual-reality headset fails to provide an answer, and limits instead the breadth of data available to perceive the movements’ expressive scope.
36To bring out the shadow-avatar’s expressiveness, the mocaptor must resort to an operation comparable to the phenomenon of empathy, described by Couchot as the combination of “an emotional resonance and a motor resonance that is automatically triggered, uncontrollable and unintentional, and which produces, though not systematically, a sharing of another’s motor and emotional states [as well as] a controlled and intentional taking of another’s subjective perspective, which we associate with self-awareness” (Couchot, 2013, p. 58). The mocaptor must allow himself to be permeated by the shadow-avatar’s feelings and expressiveness, all the while remaining aware of what his motion-capture-suit-wearing body can do. Crucially, the mocaptor’s task is to control the shadow-avatar in relation with the narrator. Visually absorbed in a form of empathetic control of the avatar, the mocaptor maintains a sonic connection with the actor’s vocal interpretation, in relation to a specific story’s narrative context. On the mixed-reality stage, the director also plays a decisive role in guiding the shadow-avatar’s expression. He develops his own relation of empathy with the virtual character in a way that lets him experience the emotions at hand and simultaneously put them at a distance, allowing him to assess their relevance with regards to the narrator’s performance. The narrator also develops his character in response to the shadow-avatar’s expressiveness. To wrap up this exploration of animation design in connection with a narrator’s performance, let us now briefly describe the method used to create a cast of five shadow-avatars in CAVOAV.
37The present example shows that AvatarStaging is a technology that takes us beyond pre-visualization, where rendering necessarily begins in post-production. Here, recorded animations are put to use in their present state and adjusted over the course of the rehearsals. This makes it for instance possible to replay a shadow-avatar’s actions, giving the mocaptor time to develop a second character’s reaction. A genuine capacity for empathetic projection onto the second shadow-avatar is needed to respond to the actions previously recorded for the first shadow-avatar. For the director, the challenge then consists in developing a different character, defined by its own gestural expressiveness. This process relies once more on a triangular relationship with the narrator, who must also find a different interpretation for the voice of the new character. The possibility of recording multiple takes of the animations on the fly is no small advantage: this makes it possible to replay them separately and to take in their expressiveness. Finding new avenues for vocal and physical interpretation, both for the actor, the mocaptor, and the director, helps improve the quality of the interactions and the credibility of the imagined theatrical situation. AvatarStaging allows for a mise-en-scène that relies on avatar direction, in a similar way to actor direction, and results in a relationship between the narrator and the five shadow-avatars. Animations are edited in real time up to the last rehearsal as a result of new propositions from the narrator, even once the mocaptor is no longer present. Because this specific show was produced in three languages, with three different narrators—based on the same animation sequences—I was also able to adjust acting intentions for the narrator and the shadow-avatars from one performance to the next.
38Lastly, the final combination of the recorded animations is a crucial step for the quality of the performance. By coupling a system of animation classification with the real-time linkage plugin AKN_Regie, I obtained a first convincing result, the description of which exceeds the scope of this article (Gagneré et al., 2020). The challenges of avatar direction nevertheless make up a complex and fascinating area of research which this project barely brushes upon.
39Our two research-creation projects tackle the concept of the avatar from two complementary standpoints. Les Double-Autres relies on a creative pipeline that provides real-time access to all the components of the photorealistic matrix-sculptures of the chosen human models. Each Double-Other provides a chance to explore the myriad facets of the artist’s and model’s visions, combined and projected onto a malleable virtual creature whose unique identity detaches and emancipates itself to engage with the audience. L’Ombre, on the other hand, draws on a creative environment in which avatars can interact with a physical actor, exist on a stage, and tell captivating stories to an audience. The avatar, dubbed “shadow-avatar,” allows the director who guides it, the mocaptor who animates it, and the actor with whom it engages, to explore new dramatic possibilities based on the spectacular resources afforded to it by virtual technology on a mixed-reality stage.
40An interesting prospect consequently emerged, giving us the idea to seize on the fine-tuned process of identity-modeling at play in the Double-Other and to direct this Double-Other on a theater stage, where stories can be told. The possibility of evolving a character in relation to dramatic situations, or exploring acting situations inspired by a given character, piqued our imagination. The interrelationship of two Double-Others capable of influencing each other and of drawing on their own expressive capabilities to mutually densify each other’s stage presence emerged as a promising avenue for research-creation. Each interactive portrait of Les Double-Autres lets the audience engage with a single avatar. In L’Ombre, the shadow-avatars are clones derived from a single silhouette. Working with Double-Others each harboring a singular identity could open up new perspectives in the field of stage direction.
Figure 7
Les Compères [The Stooges], Georges Gagneré and Cédric Plessiet’s double-others.
Graphics credit: Cédric Plessiet.
41This resulted in our idea to work our own Double-Others into a future production called Les Spectateurs [The Spectators]. The protagonists of the play, “Georges Gagneré” and “Cédric Plessiet” (fig. 7), share a virtual stage as they watch a physical actor attempt to recite Hamlet’s famous monologue (“To be or not to be, that is the question”). The performance will include several clones of each Double-Other, and will raise numerous issues related to the synchronization of live and virtual performances, with the potential for multiplied interactions.
42Finally, we should add that all of the technologies and processes used in our past and future research-creation projects are generic and can be used in other contexts, in a spirit of mutual help and sharing. The generalization of on-set pre-visualization offers a paradigm suitable for avatar-related projects, as well as for artistic iterations typical of research-creation projects. All of these creative and exploratory dynamics, and all of this technological and methodological experimentation, inspire us to pursue and consolidate this line of investigation in our upcoming research.