Navigation – Plan du site

AccueilNuméros9Démarches artistiques et création...Spectacular and interactive avata...

Démarches artistiques et créations avatariales

Spectacular and interactive avatar staging: two research-creation strategies involving real-time pre-visualization tools

Cédric Plessiet et Georges Gagneré
Traduction de Armelle Chrétien
Cet article est une traduction de :
Mises en scène spectaculaires et interactives de l’avatar. Deux approches en recherche-création utilisant des technologies de prévisualisation en temps réel [fr]

Résumé

One of the defining aspects of Cédric Plessiet and Georges Gagneré’s avatar creations is an expressive responsiveness. These two artists and researchers make use of the techniques and processes of on-set previsualization, which allows directors and performers to control avatars in real time, in a virtual environment that is nearing its final stage. In this paper, Cédric Plessiet uses his series of interactive portraits Les Double-Autres [The Double Others] to illustrate Taupipeline, a software architecture that allows him to establish a dialogue with the physical models on which his realistic avatars are based. In turn, Georges Gagneré writes about his work directing shadow avatars in his performance L’Ombre [The Shadow], based on H. C. Andersen’s homonymous tale. To conclude, the two artists talk about how their processes complete each other in their next joint research-creation project, Les Spectateurs [The Spectators].

Haut de page

Texte intégral

1The avatar is at the heart of Georges Gagneré and Cédric Plessiet’s research and artistic productions. These two artists and researchers are on the Digital Arts and Technologies teaching team, and are members of the Digital Image and Virtual Reality team within the Visual and Contemporary Arts research lab at Paris 8 University. Georges Gagneré has been exploring the role of avatars as acting partners opposite physical performers on a mixed-reality stage that brings together real and virtual worlds. Cédric Plessiet, on the other hand, focuses on the realistic digital double existence of actors—or of physical models, in the context of interactive installations. Their first collaboration, which dates back to 2015 (Gagneré & Plessiet, 2015), resulted in a fruitful cross-fertilization of their explorations into numerous research questions, such as: how can avatars be used in a way that lets them exist artistically and that brings forward new forms of dramaturgy, promoting interactions with the audience? How can we draw on real-time digital tools from animated movies and video games when designing these avatars?

2By simultaneously positioning our research in the fields of animated film, video games, visual arts, and performing arts, we observed a proliferation of definitions and uses of the avatar. This term also appears as a widely used concept in a growing number of scientific fields, including computer science disciplines dealing with language, conversational agents, cognition, and visual rendering. As a result, we must first outline our definition and use of this protean concept based on the specific computer tools used in our research-creation projects. Recent developments have led to a paradigm shift concerning these software tools, which can be linked to the emergence of increasingly high-quality methods of real-time rendering. These developments are central to our artistic practice, which aims to enable interactions between virtual avatars and the real world.

  • 1 Les Double-Autres, Microfolie Festival 2018 (Villiers-le-Bel), Exhibition Play!, 2019 (Bar-le-Duc), (...)
  • 2 L’Ombre, based on the work of H. C. Andersen. Directing, scenography, and digital programming: Geor (...)

3Consequently, we will first introduce the challenges raised by real-time technology in the context of on-set pre-visualization, a defining aspect of our work, before going on to describe our avatars and their behaviors. Secondly, we have chosen to present two separate but parallel research-creation projects: Les Double-Autres [The Double Others],1 a collection of interactive portraits designed by Cédric Plessiet, and L’Ombre [The Shadow],2 a performance directed by Georges Gagneré. Lastly, the description of a new joint research-creation project will demonstrate the convergence of our investigations.

On-set pre-visualization and avatars

The generalization of on-set pre-visualization

4On-set pre-visualization is an emergent response to the needs of the film industry and the video game industry, as a way to improve the technological and artistic integration of digital technology. As a tool for the production of audiovisual content, it provides movie directors and producers with an immediate rendering of images that will come close to the end result intended for an audience. This means that the difference between so-called pre-rendered productions (typically movies, which are made up of image sequences in which each sequence requires extensive computation time) and “real-time” productions (typically video games, because each image is the result of near-instant calculation) tends to fade. The film industry in particular has been affected by the new architecture of many software processes and tools designed to meet production needs. These “pipelines” make it possible to capture and arrange images in real time (Antoniades, 2016) thanks to hyper-realistic digital doubles of existing human individuals (Seymour, 2017). One of the aims of pre-visualization technology is to help physical performers understand the virtual environments (including special effects and virtual creatures) inhabited by their virtual characters (Bouville, 2016).

5The tools and methods of on-set pre-visualization can be used for any kind of artistic practice that calls for physical/digital hybridization processes, during the stages of design, production, and exhibition, and can also appeal to disciplines that lie outside the fields of film and video games, such as live performance, visual arts, or music. Our collaborative projects make active use of on-set pre-visualization, even at the final stage when presenting the work to the public. The tools we use are not only stepping stones used to approximate the end result: they are an integral part of the production presented to the audience. We will nevertheless continue to use the term “pre-visualization” in the context of our research-creation projects because it encompasses the real-time dimension of the processes implemented as well as their existence on a timeline that extends from the initial stage of design to that of production, all the way to finalization. Furthermore, these artistic productions systematically involve avatars.

Avatar definitions

6What do we mean by the word avatar? The term shows up in numerous contexts and is often used as a synonym for virtual actor. Among these myriad definitions, we are particularly interested in the term as it first appeared in the Lucasfilm game Habitat, where it refers to a user’s interface or vehicle in the virtual world of a video game. In their pioneering paper on the emergence of new recreational behaviors (Perény & Amato, 2010), Étienne Armand Amato and Étienne Perény also provide an in-depth examination of this concept, insisting that:

Increasingly, the real, as well as the potential, will be split in two by the virtual, in the English sense of the word: that of another reality, which is simply a quasi-reality, or the reality of a virtual and operable image, operated and inter-operated [agie et inter-agie], and ultimately liable of turning of anyone and anything into an avatar that can manifest itself on a screen. (p. 112)

7An avatar is also used to describe the virtual entity that lets us visualize a process of motion capture applied to a human individual, bringing us back to the idea of transferring motion-based data from a physical body onto a digital one.

8Our avatars are informed by the notion of matrix-sculpture (Plessiet, 2019). Based on digital representations that allow users to create a generally anthropomorphic entity, the matrix-sculpture consists of a mesh of points, or polygons, making up a virtual outer envelope. Coupled with joints that collectively make up a rig, this polygon envelope can assume an infinite variety of positions in space. It provides the matrix for the corporeal and facial movements that result from the way the rig, or skeleton, is controlled across time.

Avatar classifications

9The avatar as a concept assumes various names in various fields. In the 1980s, Daniel Thalmann and Nadia Magnenat-Thalmann’s groundbreaking work led to some of the earliest uses of realistic avatars in 3D animated films (Magnenat-Thalmann, 1987), and suggested classifying virtual actors into four main categories: participant actors, controlled through motion-capture technology; guided actors, controlled through peripheral devices; autonomous actors, capable of interacting with their environment; and lastly, actors capable of interacting with other real or virtual actors. The aim of this computational approach was to create “intelligent” virtual human beings capable of simulating real human individuals (Couchot, 2013). Researchers in computer and cognitive sciences later shifted their focus to animated conversational agents. Instead of simulating human beings in full, the idea was to develop complex interactions using only their faces. Research conducted by Catherine Pelachaud (1996) in this field led to the development of GRETA (Rosis, 2003), an expressive and interactive conversational agent platform capable of communicating with users both verbally and non-verbally, by way of looks, movements of the head and chest, facial expressions, and gestures. Artificial intelligence also addresses the matter of complex interactions between multiple agents by attempting to take into account the enactive paradigm of self-organization in an environment (Varela, 1993). One example of this is the model MASCARET, developed by Pierre Chevaillier and Pierre de Loor, which is dedicated to educational experimentation (Buche, 2004).

10Other researchers have focused on ways of describing avatars’ actions by drawing on theatrical stage techniques (Ronfard, 2016), leading them to examine how emotions can be caused by neutral expressions, working to direct the facial expressions of a virtual performer in the manner of a stage director (Barbulescu, 2014). In the field of live performance, the growing presence of virtual characters has challenged our concept of what defines actors and their onstage presence (Ferral, 2012). More broadly, the need to clarify the role of virtual performers in digital art practices such as animated film and video games has led us to develop a classification system based on two criteria: origin of movement and origin of decision (Plessiet, 2019). According to whether the origin of the choice or of the movement itself is internal or external to the entity, four virtual figures stand out: the puppet, whose movements and decisions have an external source; the autonomous actor, characterized in both cases by internal causality; the golem, whose movements are self-produced but who is bereft of free will; and the mask, whose movements originate from outside itself, but whose decisions are its own. As an example, characters in the video game The Sims (Wright, 2000) can be seen as golems obeying the player; objects, on the other hand, contain action-oriented decisions that require the characters’ external movements to actualize themselves (fig. 1). These objects are virtual masks.

Figure 1

Figure 1

The “Puppet, Golem, Actor, Mask” model (PGAM).

Graphics credit: Cédric Plessiet.

11To better understand how the figure of the avatar can play a central part in artistic processes, the next part of this paper describes our use of the matrix-sculpture in two research-creation projects: Les Double-Autres, a collection of interactive portraits by Cédric Plessiet, and L’Ombre, a performance directed by Georges Gagneré (Gagneré, 2020) based on Hans Christian Andersen’s homonymous tale. The first project explores the animation of realistic virtual actors as part of an interactive installation. The second project features non-realistic avatars engaged in a hybrid relationship with an actor over a 50-minute theatrical performance. In both cases, the use of matrix-sculptures and the presentation of the artwork to the public make generalized use of on-set pre-visualization technology.

Les Double-Autres, by Cédric Plessiet

12The “Double-Other” is a virtual entity based on an actual person, the model, and which forms the basis for a gallery of interactive portraits. The avatar’s design involves a hybrid approach combining 3D digitization, artistic process, and task automation.

The Double-Other, an emancipated avatar

13First, I shall retrace the main steps by which I am able to achieve a photorealistic digital double of the physical model that my three-dimensional interactive portrait is based on. Traditional processes borrowed from the video game industry suggest several ways of creating a mesh using 3D capture systems, such as photogrammetry or a 3D scanner. My approach also separates body design from face design. I design the body using preexisting meshes that I customize with Blender and Zbrush. For the face mesh, I use a 3D scanner, obtaining a mesh that is indeed lifelike but unfortunately inoperable in its present state because it is not designed to deform. The geometry of the 3D scanner does not contain the curvature lines that allow for the type of distortions required by facial expressions. I must therefore edit its topology, so that it can later be stretched out, crushed, or curved.

14The next step consists in coupling this customized mesh with a rig that allows for the deformation and animation of the matrix-sculpture. The animation controls take into account the type of freedom of movement I want to work into the specific anatomy of the mesh derived from the physical model. To ensure consistent facial movements, I break down possible expressions according to the Facial Action Coding System or FACS (Eckman, 2002). This system relies on a limited number of facial action units (FAU) that make up so many elements of expression related to the face’s anatomy. This means that I must conduct a scan of each of the thirty or so expressions specific to the physical model which these FAUs are based on.

15With preexisting mesh elements related to the body and a score of new face elements, I must now design an avatar that should be both a highly resemblant double of the physical model and one sufficiently manipulable to allow it to perform its own expressive movements, turning it into what I call a “Double-Other.” The Double-Other is a matrix-sculpture which couples a great faithfulness to the physical model with the possibility of performing complex movements that fall in line with the different possibilities set out by the PGAM model (see section “Avatar Classifications”). The virtual Double-Other splits the physical model in two, and emancipates itself through movements of its own.

Taupipeline, the technology behind the Double-Others

16Wishing to focus on more interesting creative tasks and to obtain rapid results, I next considered how to optimize the creation of a Double-Other. This need for rapid reaction times is at the heart of on-set previsualization and makes it possible to monitor changes in the end result at each stage. It ties in with the agile philosophy (Autissier, 2021), according to which the capacity to iterate quickly on a working product allows both for creative dynamism and artistic control. These considerations led to the development of Taupipeline, which brings together various processes (modeling, rigging, animation, rendering) required to create a Double-Other.

17Taupipeline (fig. 2) crucially relies on the free software tools Tactic and Afanasy, used for asset management and rendering distribution, respectively. I use standard computer graphics software, such as Maya, Blender, and Photoshop, to produce 3D components. Practically speaking, I start by searching through existing assets on Taupipeline for components that might fit with the new physical model. If I cannot find any matches, I recreate or edit new components which I then add to the database. For the face, I need to design the thirty or so FAUs required for the facial expressions. I use a process automation software that I programmed which saved me several hours of repetitive, non-creative, and tedious work. Automation also makes it easier to avoid manipulation errors. Finally, the Tauporchestre module carries out this set of automated tasks on several computers at once. Any change brought to a specific element is thus easily and rapidly computed and applied to each and every one of the matrix-sculpture’s components.

Figure 2

Figure 2

Taupipeline and Taupemodel modules.

Graphics credit: Cédric Plessiet.

18Taupipeline represents a compromise between 3D character design solutions like Metahuman (Metahuman, 2021), which provide immediate but stereotypical results, and the ex nihilo design of a 3D character with a more personal feel, which takes a very long time and requires an elaborate skillset.

Creative process and artistic results

19Animations and interactions are then finalized thanks to the virtual-reality and motion-capture processes brought together in the Taupemodel module (fig. 2), which consists of a motion-capture system and the real-time game engine Unreal Engine. This engine was chosen because of its graphic capabilities, especially for skinning, and because of the easy access to its source code, which allows for advanced modifications. Moreover, the native integration of third-party interface software makes it easier to associate with my own technology.

20Equipped with a motion-capture device connected to the video monitor and a virtual reality headset (fig. 3a), I strike a pose or perform whatever motion I wish to pass on to the Double-Other. I can then study it from an external point of view, and manipulate the rig to correct potential animation issues (a collision between body and limb, the position of the fingers, etc.). I then adjust the different components of the scene, the lighting, and the camera frame. All of these operations are done in virtual reality, which allows me to have a clear and tangible understanding of volumes and of the 3D render. By living in the skin of the Double-Other I wish to animate, I enter into a relationship of empathy, even though the avatar is for the most part nothing like me (fig. 3b).

Figure 3b

Figure 3b

(a) Cédric Plessiet in Taupemodel; (b) Eden Weiss’s Double-Other, controlled in real time by Cédric Plessiet in Taupemodel.

Graphics credit: Cédric Plessiet.

21The interactive portrait Ça bug dans les rotations [A Bug in the Rotations] features one of Georges Gagneré’s Double-Others (fig. 4, left), his head subjected to untimely rotations which the viewer can attempt to rectify (or intensify) by operating a rotary switch. The portrait harks back to conversations raised by rotation issues during a previous collaboration, in relation to a virtual puppet. La Geôle [The Jail] explores the autonomy of my own Double-Other (fig. 4, right). This self-portrait in-the-making shows me writing a book. I first started working on it while I was in the process of writing my dissertation. My Double-Other develops his own writing routine through a process of trial-and-error in an absurd universe. The learning pattern relies, among other things, on the use of artificial neural networks.

Figure 4

Figure 4

Left, Georges Gagneré’s Double-Other in Ça bug dans les rotations; right, Cédric Plessiet’s Double-Other in La Geôle, with visible decision-making operations.

Graphics credit: Cédric Plessiet.

22This accelerated avatar production process allows for rapid and inexpensive changes. The increased speed and flexibility make it possible to bring changes to the design method, for instance by inviting the original human model back into the process, letting her engage with me, give her opinion, request adjustments, or explore her virtual body. Some models have wanted to change their hairstyles, others have wanted to see themselves pregnant, etc. The dialogue between the model and the artist is made possible by Taupipeline. The Double-Other becomes a co-creation in which the virtual body balances my vision of the model with her own self-perception and the way she wishes to avatarize herself.

Staging the avatar, by Georges Gagneré

23I will now describe my use of avatars in the research-creation project L’Ombre (Gagneré, 2020), starting with the aesthetic framework of the performance and the avatar control system. I will then examine the artistic questions raised by matters of stage direction, as well as the solutions developed to animate the avatars and work the animations into the actor’s performance.

CAVOAV and AvatarStaging

  • 3 The French word castelet refers to a component of puppet theater scenery which creates a frame arou (...)

24The show uses the concept of CAstelet Virtuel d’OmbrAVatars (Virtual Puppet Booth for Shadow-Avatars, CAVOAV) (Gagneré & Ternova, 2020) which borrows from traditional shadow play theater and the dramaturgical possibilities afforded by the figure of the shadow, drawing particular inspiration from Ernst Theodor Amadeus Hoffmann (1776-1822), Adelbert von Chamisso (1781-1838), and Hans Christian Andersen (1805-1875). This approach lets me breathe three-dimensional life into shadow-like avatars, dubbed ombravatars [shadow-avatars] (fig. 5). Shadow-avatars are matrix-sculptures whose mesh consists of a two-dimensional human silhouette replicating the exact shadow cast by an individual when he or she is lit up by a spotlight positioned perpendicular to a screen. The mesh is fitted with a rig, which has the specificity of belonging to a plane. This “flat” rig nevertheless allows for a three-dimensional manipulation of the silhouette thanks to motion-capture devices that make it possible to transpose the boundless range of a person’s natural movements. CAVOAV is a three-dimensional space encased in a virtual frame. Projected onto a stage, the space opens up a kind of theater within a theater, populated by shadow-avatars that can engage with the physical actors performing in front of the screen. Thanks to the real-time engine’s 3D rendering, the visual effect conjures up the existence of a virtual puppet booth3 set up behind the rigid virtual frame of the image.

Figure 5

Figure 5

Excerpt from the performance L’Ombre, directed and programmed by Georges Gagneré.

Graphics credit: didascalie.net.

25A shadow-avatar is an anthropomorphic entity capable of existing both as a shadow, projected in two dimensions, and as a three-dimensional silhouette, as shown in Figure 5: a shadow-avatar (in gray) is shown acting opposite its own shadow cast against the windows (in black). Like any virtual entity, shadow-avatars can lend themselves to any special effect imaginable and programmable. The animation of the silhouettes is based on AvatarStaging4 (Gagneré, 2018), a system which combines several software tools in order to transpose movements performed by an actor equipped with a motion-capture device—the “mocaptor”—onto the avatars inside the 3D digital space CAVOAV, which represents the virtual component of the mixed-reality stage.

Figure 6

Figure 6

AvatarStaging.

Graphics credit: Georges Gagneré.

26Generically speaking, AvatarStaging brings together three processes, successively detailed in each of the next three paragraphs. They let a mocaptor control an avatar and develop a relationship between said avatar and a physical actor on a mixed-reality stage (A + B). Figure 6 is a schematic representation of how these three processes operate for a mocaptor, an avatar, and an actor, variations of which can be inferred in more complex configurations. In the first process, the mocaptor is wearing a full-body suit, which can be optical or inertial, and which makes it possible to collect data relating to the position of the sensors situated on specific areas of the mocaptor’s morphology (fig. 6, Zone C). The motion-capture data is then transferred to a first specific matrix-sculpture, i.e., a virtual skeleton inferred from the position of the sensors situated on the mocaptor’s outer envelope. This skeleton is hence the result of a mathematical approximation and does not match the mocaptor’s actual posture or frame. Moreover, multiple parasitic factors affecting data capture can cause distortions. Their impact on the artistic quality of the movements must constantly be assessed and corrected, if necessary.

27The second process, called motion retargeting, involves transposing the movements performed by the mocaptor’s first virtual skeleton onto the virtual skeleton of the matrix-sculpture corresponding to the avatar chosen to perform in the 3D space opposite the actor. In L’Ombre, the shadow-avatar has a very specific build, inevitably distorting the mocaptor’s original motions, an aspect which I will discuss at greater length in relation to the project’s artistic outcomes (see section “Methodology of Animation Design”).

28The third and last process consists in organizing the positions of the avatar and/or its prerecorded animations according to the digital setting and to the positions of the physical actor (fig. 6, zone B), using the AKN_Regie plugin (which I programmed in a game engine).

29To sum up: the mocaptor present in space C develops a stage relationship to the actor on stage A thanks to the video playback monitors placed around him, which enable him to see the result of his movements projected onto his avatar in digital space B—a dynamic made possible by the three processes of motion capture, motion retargeting, and motion control with AKN_Regie.

Artistic intention behind the avatars

30The dramatic hypothesis underlying L’Ombre goes as follows: a physical actor narrates Andersen’s homonymous tale to a physical audience and to shadow-avatars inside CAVOAV. Over the course of the narration, the shadow-avatars begin to reenact a series of scenes, taking on the part of five protagonists from this fantastic tale about a scholar from the North who leaves his shadow behind in some torrid land of the South. Certain parts of the story contain dialogue, so that the narrator performs the characters’ voices in turn; their movements, however, originate from other sources. One might picture a group of five mocaptors controlling the five shadow-avatars from behind the scenes with the use of AvatarStaging, while a physical narrator is narrating the story. But because I wished to achieve a light-weight form which could be performed in any place equipped with a projection surface, I wanted the actor to be able to man the technical aspects on his own, in full view of the audience (visual animations, music score, light and sound effects). In other words, the dramatic hypothesis meant that the actor had to be able to activate the prerecorded animations that simulate attentiveness and active participation from the shadow-avatars. This led to a first problem: reconciling the actor’s freedom of interpretation—in terms of pacing and acting intentions—with prerecorded animations, which imply firmly set intentions and a predetermined pace. A second challenge was to record lifelike animations capable of endowing the shadow-avatars with some kind of stage presence. We shall first examine how to achieve motion captures capable of breathing life into the shadow-avatars acting out the narrator’s story. Failing to achieve lifelike animations would have ruled out the possibility of combining them with the narrator’s live performance in a consistent way. These mechanisms will be briefly discussed further on.

31In film and audiovisual media, motion-capture animation usually involves putting the mocaptor in the same physical situation as the one their avatar will be performing in, before carrying out the three operations depicted above to design the corresponding animation in post-production. This approach involves a separation between the production of the movements on the one hand, and the reception of the animated avatar’s expressiveness by fellow performers on the other. This means that animations cannot be modified to take into account the kind of mutual influence that informs human interactions.

32Picture now our mocaptor stepping into the part of a shadow-avatar on the mixed-reality stage. The narrator is performing opposite another actor dressed in a motion-capture suit, in no way his final acting partner. A fundamental aspect of my approach involves making use of on-set previsualization to address the aesthetic nature of the shadow-avatar. More important than the mocaptor’s motions per se are these movements’ effects on the shadow-avatar, the narrator’s final acting partner. Over the course of the rehearsals and of the recording of the animations, AvatarStaging makes it possible for the physical actor to step into his role (fig. 6) and to perform on the final mixed-reality stage opposite the shadow-avatar. Conversely, the mocaptor uses video monitors to watch the results of the interactions and to adjust them in real time. In a traditional theater production, the possibility of working in final performance conditions often only comes up at the very end, a few days before dress rehearsal, when the costumes, set, lights, and special effects are finally brought together and adjusted. My approach consists in bringing all the components together from the start and making whatever changes are needed on the spot, over the course of the rehearsals.

33Next, the narrator and the shadow-avatar must develop their stage relationship under the director’s watch. Indeed, the director is also in charge of writing the dramatic hypothesis which determines how the shadow-avatars will listen to the story. In this context, a side note about how the theater traditionally addresses character-building seems necessary. Generally speaking, the challenge for any actor is to be able to control their emotions in order to develop them according to a dramatic intent co-elaborated with the stage director and acting partners, based on the material provided by the playwright. Actors must paradoxically balance control and honesty to develop their characters. In other words, they must simultaneously inhabit their own body and distance themselves from their emotions, in order to be able to arrange them in a way that brings out a character for the audience. In L’Ombre, the narrator’s acting technique falls in line with this theatrical tradition. But the mocaptor’s character inhabits a body that isn’t his own, hence disrupting the traditional relationship. Up to now, the relation to the avatar character has received little attention from theater studies (Gagneré, 2015; Kennedy, 2021). I will examine some of its aspects as explored in collaboration with the mocaptor who carried out the recordings: an experienced drama student who had already taken part in several motion-capture projects involving an inertial suit.

34Beyond the specifics of character development, the entities in relation are of different natures. Human beings do not naturally coexist with shadow-avatars. The stage director must assess the nature of this relationship in order to develop a convincing dramaturgy. Because of this, I chose to add a silent introduction to the performance: I suggested that the physical narrator touch the shadow-avatars with his own shadow cast by the light of the digital projector. The idea was to conjure up a situation where roaming shadows are progressively tamed and invited to share a moment with the audience. What I like about the nature of the shadow is precisely its ambivalent relation to human corporeality. The shadow is an extension of our presence in the world which also prompts us to reflect on its vulnerability. Are we alone with ourselves? Who is this other self, copying my motions to perfection? What are its powers? The shift from 2D to 3D afforded by the shadow-avatar is a way of moving the viewer to accept the fact that the narrator rubs shoulders with shadows who transform into bizarre creatures. The emergence of triangular stage relationships between the actor, the mocaptor, and the director requires iterative adjustments over the course of the rehearsals. This allows all the partners involved to improve their technique in real time, in a situation identical to the one experienced by the audience.

Methodology of animation design

35Let us now turn to the mocaptor/shadow-avatar’s specific acting technique. The aim of the first process is to acquire command of the motion-capture device. This calls for constant attention and evaluation, as multiple parameters can affect the virtual skeleton obtained through motion capture. Furthermore, because of the inertial suit used in L’Ombre, the sensors’ sensitivity to magnetic fields induced small but recurrent distortions which called for continued vigilance on the mocaptor’s part. Next, the specific constraints of the motion-retargeting process must be taken into account: what will the relevant expressive movements turn out to be after their transposition to the specific morphology of the shadow-avatar? Among the many adjustments required to balance possible distortions or confusions, we found: bending the arms to bring life to the silhouette’s forced dimensionality; positioning the arms alongside the body to avoid collisions with the hands; emphasizing signs of global body orientation, such as an arm stretched forward, to make it clear where the shadow-avatar is looking and make up for ambiguity caused by the absence of depth; as well as designing the avatar’s walk to identify emotions by its way of moving through space. Developing a proprioceptive sense of the shadow-avatar’s envelope, that is to say a satisfying integration of the body’s perceptions, is a challenging task for the mocaptor. The apparent solution which consists in wrapping the mocaptor inside this envelope thanks to a virtual-reality headset fails to provide an answer, and limits instead the breadth of data available to perceive the movements’ expressive scope.

36To bring out the shadow-avatar’s expressiveness, the mocaptor must resort to an operation comparable to the phenomenon of empathy, described by Couchot as the combination of “an emotional resonance and a motor resonance that is automatically triggered, uncontrollable and unintentional, and which produces, though not systematically, a sharing of another’s motor and emotional states [as well as] a controlled and intentional taking of another’s subjective perspective, which we associate with self-awareness” (Couchot, 2013, p. 58). The mocaptor must allow himself to be permeated by the shadow-avatar’s feelings and expressiveness, all the while remaining aware of what his motion-capture-suit-wearing body can do. Crucially, the mocaptor’s task is to control the shadow-avatar in relation with the narrator. Visually absorbed in a form of empathetic control of the avatar, the mocaptor maintains a sonic connection with the actor’s vocal interpretation, in relation to a specific story’s narrative context. On the mixed-reality stage, the director also plays a decisive role in guiding the shadow-avatar’s expression. He develops his own relation of empathy with the virtual character in a way that lets him experience the emotions at hand and simultaneously put them at a distance, allowing him to assess their relevance with regards to the narrator’s performance. The narrator also develops his character in response to the shadow-avatar’s expressiveness. To wrap up this exploration of animation design in connection with a narrator’s performance, let us now briefly describe the method used to create a cast of five shadow-avatars in CAVOAV.

37The present example shows that AvatarStaging is a technology that takes us beyond pre-visualization, where rendering necessarily begins in post-production. Here, recorded animations are put to use in their present state and adjusted over the course of the rehearsals. This makes it for instance possible to replay a shadow-avatar’s actions, giving the mocaptor time to develop a second character’s reaction. A genuine capacity for empathetic projection onto the second shadow-avatar is needed to respond to the actions previously recorded for the first shadow-avatar. For the director, the challenge then consists in developing a different character, defined by its own gestural expressiveness. This process relies once more on a triangular relationship with the narrator, who must also find a different interpretation for the voice of the new character. The possibility of recording multiple takes of the animations on the fly is no small advantage: this makes it possible to replay them separately and to take in their expressiveness. Finding new avenues for vocal and physical interpretation, both for the actor, the mocaptor, and the director, helps improve the quality of the interactions and the credibility of the imagined theatrical situation. AvatarStaging allows for a mise-en-scène that relies on avatar direction, in a similar way to actor direction, and results in a relationship between the narrator and the five shadow-avatars. Animations are edited in real time up to the last rehearsal as a result of new propositions from the narrator, even once the mocaptor is no longer present. Because this specific show was produced in three languages, with three different narrators—based on the same animation sequences—I was also able to adjust acting intentions for the narrator and the shadow-avatars from one performance to the next.

38Lastly, the final combination of the recorded animations is a crucial step for the quality of the performance. By coupling a system of animation classification with the real-time linkage plugin AKN_Regie, I obtained a first convincing result, the description of which exceeds the scope of this article (Gagneré et al., 2020). The challenges of avatar direction nevertheless make up a complex and fascinating area of research which this project barely brushes upon.

Future directions

39Our two research-creation projects tackle the concept of the avatar from two complementary standpoints. Les Double-Autres relies on a creative pipeline that provides real-time access to all the components of the photorealistic matrix-sculptures of the chosen human models. Each Double-Other provides a chance to explore the myriad facets of the artist’s and model’s visions, combined and projected onto a malleable virtual creature whose unique identity detaches and emancipates itself to engage with the audience. L’Ombre, on the other hand, draws on a creative environment in which avatars can interact with a physical actor, exist on a stage, and tell captivating stories to an audience. The avatar, dubbed “shadow-avatar,” allows the director who guides it, the mocaptor who animates it, and the actor with whom it engages, to explore new dramatic possibilities based on the spectacular resources afforded to it by virtual technology on a mixed-reality stage.

40An interesting prospect consequently emerged, giving us the idea to seize on the fine-tuned process of identity-modeling at play in the Double-Other and to direct this Double-Other on a theater stage, where stories can be told. The possibility of evolving a character in relation to dramatic situations, or exploring acting situations inspired by a given character, piqued our imagination. The interrelationship of two Double-Others capable of influencing each other and of drawing on their own expressive capabilities to mutually densify each other’s stage presence emerged as a promising avenue for research-creation. Each interactive portrait of Les Double-Autres lets the audience engage with a single avatar. In L’Ombre, the shadow-avatars are clones derived from a single silhouette. Working with Double-Others each harboring a singular identity could open up new perspectives in the field of stage direction.

Figure 7

Figure 7

Les Compères [The Stooges], Georges Gagneré and Cédric Plessiet’s double-others.

Graphics credit: Cédric Plessiet.

41This resulted in our idea to work our own Double-Others into a future production called Les Spectateurs [The Spectators]. The protagonists of the play, “Georges Gagneré” and “Cédric Plessiet” (fig. 7), share a virtual stage as they watch a physical actor attempt to recite Hamlet’s famous monologue (“To be or not to be, that is the question”). The performance will include several clones of each Double-Other, and will raise numerous issues related to the synchronization of live and virtual performances, with the potential for multiplied interactions.

42Finally, we should add that all of the technologies and processes used in our past and future research-creation projects are generic and can be used in other contexts, in a spirit of mutual help and sharing. The generalization of on-set pre-visualization offers a paradigm suitable for avatar-related projects, as well as for artistic iterations typical of research-creation projects. All of these creative and exploratory dynamics, and all of this technological and methodological experimentation, inspire us to pursue and consolidate this line of investigation in our upcoming research.

Haut de page

Bibliographie

Antoniades, T. (2016). Creating a live real-time performance-captured digital human. In ACM SIGGRAPH 2016 Real-Time Live! (SIGGRAPH ‘16). Association for Computing Machinery. https://doi.org/10.1145/2933540.2933551

Autissier, D. & Moutot J.-M. (2015). Le changement agile. Se transformer rapidement et de manière durable. Dunod.

Barbulescu, A., Ronfard, R., Bailly, G., Gagneré, G., & Cakmak, H. (2014). Beyond basic emotions: Expressive virtual actors with social attitudes. In Proceedings of 7th International ACM SIGGRAPH Conference on Motion in Games 2014 (MIG 2014) (p. 39-47). Association for Computing Machinery https://doi.org/10.1145/2668064.2668084

Bouville, R., Gouranton, V., & Arnaldi, B. (2016). Virtual reality rehearsals for acting with visual effects. Proceedings of the 42nd Graphics Interface Conference (GI ‘16) (p. 125-132). Canadian Human-Computer Communications Society.

Buche, C., Querrec, R., De Loor, P., & Chevaillier, P. (2004). MASCARET: A pedagogical multi-agent system for virtual environments for training. International Journal of Distance Education Technologies (IJDET), 2(4), 41-61. http://doi.org/10.4018/jdet.2004100103

Couchot, E. (2013). Tout un peuple dans un monde miroir. In R. Bourassa & L. Poissant (eds.), Personnage virtuel et corps performatif. Effets de présence. Presses de l’Université du Québec.

Ekman, P. & Friesen, W. V. (1978). Manual for the facial action coding system. Consulting Psychologists Press.

Féral, J. (ed.). (2012). Pratiques performatives. Body Remix. Presses universitaires de Rennes/Presses de l’Université du Québec.

Gagneré, G. (2020). The shadow. In Proceedings of the 7th International Conference on Movement and Computing (MOCO ‘20), Association for Computing Machinery. https://doi.org/10.1145/3401956.3404250

Gagneré, G., Mays, T., & Ternova, A. (2020). How a hyper-actor directs avatars in virtual shadow theater. In Proceedings of the 7th International Conference on Movement and Computing (MOCO ‘20). Association for Computing Machinery. https://doi.org/10.1145/3401956.3404234

Gagneré, G. & Plessiet, C. (2015). Traversées des frontières. In H. Hachour, N. Bouhaï, & I. Saleh (eds.), Frontières numériques & artéfacts (p. 9-35). L’Harmattan.

Gagneré, G. & Plessiet, C. (2018). Experiencing avatar direction in low cost theatrical mixed reality setup. In Proceedings of the 5th International Conference on Movement and Computing (MOCO ‘18). Association for Computing Machinery. https://doi.org/10.1145/3212721.3212892

Gagneré, G. & Plessiet C. (2022). Quand le jeu vidéo est le catalyseur d’expérimentations théâtrales (2014-2019). In C. Devès (ed.), Le jeu vidéo, au carrefour de l’histoire, des arts et des médias. Les Éditions du CRHI.

Gagneré, G. & Ternova, A. (2020). CAstelet in Virtual reality for shadOw AVatars (CAVOAV). In S. Richir (ed.), Proceedings of Virtual Reality International Conference (VRIC) (p. 129-136). https://doi.org/10.20870/IJVR.2020...3316

Kennedy, J. (2021). Acting and its Double. A Practice-Led Investigation of the Nature of Acting Within Performance Capture [doctoral thesis, Auckland University of Technology]. http://hdl.handle.net/10292/13943

Magnenat-Thalmann, N. & Thalmann, D. (1987). The direction of synthetic actors in the Film Rendez-Vous a Montreal. IEEE Computer Graphics and Applications, 7, 9-19. https://doi.ieeecomputersociety.org/10.1109/MCG.1987.276934

MetaHuman. (2021). Epic Games. https://www.unrealengine.com/en-US/metahuman-creator

Pelachaud, C., Badler, N. I., & Steedman M. (1996). Generating facial expressions for speech. Cognitive Science, 20(1), 1-46. https://doi.org/10.1207/s15516709cog2001_1

Perény, É. & Amato, É. A. (2010). L’heuristique de l’avatar: polarités et fondamentaux des hypermédias et des cybermédias. Revue des interactions humaines médiatisées (RIHM), Journal of Human Mediated Interactions, 11(1), 87-115.

Plessiet, C., Gagneré, G., & Sohier, R. (2019). A proposal for the classification of virtual character. In Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications – HUCAPP (p. 168-174). https://doi.org/10.5220/0007569401680174

Plessiet, C. (2019). Quand la marionnette coupe ses fils. Recherches sur l’acteur virtuel. Université Paris 8.

Plessiet, C. (2019). De la sculpture matrice à l’acteur virtuel. Habilitation à diriger les recherches. Université Paris 8.

Ronfard, R. (2016). Notation et reconnaissance des actions scéniques par ordinateur. In M. Martinez & S. Proust (eds.), La notation du travail théâtral. Du manuscrit au numérique (p. 189-204). Lansman.

Rosis, F. de, Pelachaud, C., Poggi, I., Carofiglio, V., & Carolis, B. de (2003). From Greta’s mind to her face: Modelling the dynamics of affective states in a conversational embodied agent. International Journal of Human-Computer Studies, 59(1-2), 81-118. https://doi.org/10.1016/S1071-5819(03)00020-X

Seymour, M., Evans, C., & Libreri, K. (2017). Meet Mike: epic avatars. ACM SIGGRAPH 2017 VR Village (SIGGRAPH ‘17). Association for Computing Machinery. https://doi.org/10.1145/3089269.3089276

Varela, F., Thompson, E., & Rosch, E. (1993). L’inscription corporelle de l’esprit. Sciences cognitives et expérience humaine. Seuil.

Wright, W. (2000). The Sims. Studio Maxis, Electronic Arts.

Haut de page

Notes

1 Les Double-Autres, Microfolie Festival 2018 (Villiers-le-Bel), Exhibition Play!, 2019 (Bar-le-Duc), Exhibition Recto Verso, 2019 (Laval).

2 L’Ombre, based on the work of H. C. Andersen. Directing, scenography, and digital programming: Georges Gagneré. Performance: Eric Jakobiak (France) and Pavlo Chirva (Ukraine). Music: Tom Mays. Assistant scenographer and digital programmer: Anastasiia Ternova. Motion capture: Remy Gorski. Production: didascalie.net. Created in September 2019 in Kharkiv.

3 The French word castelet refers to a component of puppet theater scenery which creates a frame around the stage.

4 See: http://avatarstaging.eu/ (accessed February 14, 2022).

Haut de page

Table des illustrations

Titre Figure 1
Légende The “Puppet, Golem, Actor, Mask” model (PGAM).
Crédits Graphics credit: Cédric Plessiet.
URL http://journals.openedition.org/hybrid/docannexe/image/2944/img-1.png
Fichier image/png, 67k
Titre Figure 2
Légende Taupipeline and Taupemodel modules.
Crédits Graphics credit: Cédric Plessiet.
URL http://journals.openedition.org/hybrid/docannexe/image/2944/img-2.png
Fichier image/png, 116k
Titre Figure 3a
URL http://journals.openedition.org/hybrid/docannexe/image/2944/img-3.png
Fichier image/png, 404k
Titre Figure 3b
Légende (a) Cédric Plessiet in Taupemodel; (b) Eden Weiss’s Double-Other, controlled in real time by Cédric Plessiet in Taupemodel.
Crédits Graphics credit: Cédric Plessiet.
URL http://journals.openedition.org/hybrid/docannexe/image/2944/img-4.png
Fichier image/png, 235k
Titre Figure 4
Légende Left, Georges Gagneré’s Double-Other in Ça bug dans les rotations; right, Cédric Plessiet’s Double-Other in La Geôle, with visible decision-making operations.
URL http://journals.openedition.org/hybrid/docannexe/image/2944/img-5.png
Fichier image/png, 431k
Titre Figure 5
Légende Excerpt from the performance L’Ombre, directed and programmed by Georges Gagneré.
URL http://journals.openedition.org/hybrid/docannexe/image/2944/img-6.png
Fichier image/png, 3,7M
Titre Figure 6
Légende AvatarStaging.
Crédits Graphics credit: Georges Gagneré.
URL http://journals.openedition.org/hybrid/docannexe/image/2944/img-7.png
Fichier image/png, 924k
Titre Figure 7
Légende Les Compères [The Stooges], Georges Gagneré and Cédric Plessiet’s double-others.
Crédits Graphics credit: Cédric Plessiet.
URL http://journals.openedition.org/hybrid/docannexe/image/2944/img-8.png
Fichier image/png, 1,2M
Haut de page

Pour citer cet article

Référence électronique

Cédric Plessiet et Georges Gagneré, « Spectacular and interactive avatar staging: two research-creation strategies involving real-time pre-visualization tools »Hybrid [En ligne], 9 | 2022, mis en ligne le 30 novembre 2022, consulté le 03 février 2023. URL : http://journals.openedition.org/hybrid/2944 ; DOI : https://doi.org/10.4000/hybrid.2944

Haut de page

Auteurs

Cédric Plessiet

Cédric Plessiet is an associate professor with the Digital Images and Virtual Reality team (INREV) at Paris 8 University. His current research deals with filmmakers’ use of on-set previsualization and with semi-autonomous virtual actors, which allows him to work in numerous disciplines, including motion capture, real-time engines, and virtual reality. As a digital artist, Cédric Plessiet incorporates the devices he develops into his artistic creations, which have been shown at a number of festivals.

Georges Gagneré

Georges Gagneré is a stage director, digital artist, and researcher. He has been working with the platform didascalie.net since 2004, and has put together a number of performances and research-creation projects bringing together theater institutions and research laboratories. He is also a senior lecturer on the INREV team. Website: http://gagnere.fr

Haut de page

Droits d’auteur

Tous droits réservés

Haut de page
Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search