- 1 Edmond Couchot’s posthumous publication will not be taken into account in this study, given the t (...)
1 Avatarization is a way to hybridize with a digital being in computer-generated images, and we can imagine as many types of avatarization as there are individuals or technical setups. In order to browse this broad spectrum, we will examine Couchot’s approach, which offers a theory of hybridization with technological artifacts, such as avatarization. Couchot also thought about several forms of collaborations with artificial beings. We will synthesize this questioning, which led to a full-blown system, theorized between 2012 and 2018,1 based on a reflection about digital arts that started in the 1980s. Which new modes of perception come from the emergence of digital technologies? Are all avatarizations with virtual humans alike? This retrospective approach allows us to introduce key concepts in Couchot’s work: hybridization, technaesthetic experience and avatarization. Using a projective approach, we will build on this questioning and these concepts to explore them in artistic contexts, with a research-creation process.
2 Our artistic questioning focuses on the plasticity of hybridization as defined by Couchot, and on the possibility to control several avatars, or one avatar controlled by several individuals (co-avatarization). How avatarization can be used in the artistic context? What are the characteristics of the plasticity of avatarization? How to interpret avatarization when several people share the same avatar? How can Couchot’s system help understand it? How to theorize the managing of several avatars at the same time? What protocol should be followed?
- 2 We use the term “plastic” from the artistic field, in order to emphasize artistic and visual appl (...)
3 Our main hypothesis maintains that there is not a single form of avatarization, but several, which can be discriminated by plastic aspects that can allow artistic expression.2 In this sense, we agree with Etienne Pereny’s definition of global avatarization: “the avatar, by becoming mainstream, could produce a new iconic and digital relationship between man and the world” (Amato & Pereny, 2013, p. 37-59). Exploring virtual universes is made possible by avatars, and the phenomenon is questioned visually by digital artists. We firmly adhere to this thesis because we believe that the modes of avatarization can also be considered as an artistic expression per se, apart from exploring virtual worlds or from these various visual representations. Moreover, co-avatarization seems to be a powerful form of expression for interactive narration, considered as a means to structure a story around a particular interactive experience.
- 3 Couchot offers a new interpretation of Simondon, notably his “interindividual coupling” between h (...)
- 4 This work is notably carried out as part of the ArTeC University Research School project “Genealo (...)
4 The metaphor of hybridization runs through Couchot’s work and allows us to understand the changes in society brought about by digital technology, particularly in the context of artistic creation and reception. It is a way of questioning the human-machine relationship, building upon the theories of Simondon and cybernetics.3 This metaphor gradually becomes a core concept reinterpreting artistic collage, but more specifically aiming the field of interactions between living and artificial (Couchot, 1998, p. 144). Similarly, Couchot extends McLuhan’s analysis of the media (1968) by reflecting on the presence of a technological tool in the environment, because it “introduces the outside into the inside, it acts by way of incrustation” (Couchot, 1988, p. 113). Thus, hybridization is always positioned in the field of experience and therefore in action, or more precisely in the relationship between the realm of perception, the body and its environment. If we examine the corpus of books and articles Couchot wrote between 1984 and 2018,4 he uses the term “hybridization” more often than “hybrid”: it is always an ongoing process, something in the making, almost never a stable and complete state. As a result, hybridization is characterized by an evolving state, going from human to artificial and from artificial to human.
- 5 Couchot, 1988, p. 232-237.
5 How can this concept of hybridization make it possible to study digital images? In a concluding note from 1988,5 Couchot describes what he calls a “culture of hybridization,” distinguishing three ways hybridization could be applied to digital images.
1. “[O]bjects resulting from multiple and paradoxical hybridizations. hybridization first of all between the very elements forming an image” (Couchot, 1988, p. 234). Since in the digital world all data are of the same encrypted nature, everything can potentially be mixed, cut or melted with everything in order to obtain a new image (or sound).
2. “Then hybridization between modes of picture generation and modes of perception and socialization of given image. Interactivity gives digital images a capacity that no other image has ever had before, that of reacting to the beholder” (Couchot, 1988, p. 235). For Couchot, digital images are, in essence, always interactive. It’s a new mode of perception, which can only happen through hybridization with the machine, and via interactivity in particular.
- 6 He co-founded the Arts and Technologies of Image (ATI) department, with this fundamental principl (...)
3. “Lastly hybridization between symbol and tool. […] Computers enable what may be the most promising hybridization, that of figurative thought (and its fantasy) and logical-mathematical languages” (Couchot, 1988, p. 235). Couchot celebrates the possibility of exploring all the capacities of symbolic thought and technical thought.6 He refused to see an antagonism between art and science since the digital era is bound to promote the development and blossoming of an art/science dialog.
6 With this concept, Couchot questions our perception and our way of thinking with digital technologies. In order to understand more accurately our mode of perception, he suggests to study what he calls the “technaesthetic” experience: in other words, “an experience of perception in the technical act [which] constitutes a kind of perceptive habitus, of knowledge of the senses, shared by all members of a society and impacting their ways of being, acting, and thinking, but not through the channels of speech nor symbolic thought” (Couchot, 1988, p. 14). Now, this experience is essentially perceptive, using a specific way of being, “a state in which the ‘I’ is absent” (p. 8). That is why Couchot makes a distinction between the I-subject and the we-subject, thus taking up Merleau-Ponty’s idea: “Perception always belongs to the “we” (1945, p. 277). According to him, “when we use a technical means, there are always the remains of some existence already lived by others” (Couchot, 1998, p. 9). There is a particular kind of pleasure in putting aside one’s subjectivity and embracing virtual reality interactively, which is the approach of avatarization we will develop later on.
7 In his book La nature de l’art (2012), Couchot introduces the notion of “naturalization of art” in order to examine possible relationships between cognitive science, neurology, aesthetics and arts. Theses coming from these new fields resonate with his approach of hybridization and technaesthetic experience because “cognition may not be a simple mirror of the world surrounding us, but the embodiment of an action” (Couchot, 2012, p. 11). Building on Jeannerod’s theses (2002), the “naturalization of art” considers mind and thought as natural objects, contrary to a dichotomy separating “the natural world of physics and biology on one side and the ineffable and spiritual world of culture, beliefs and arts on the other” (Couchot, 2012, p. 7). Hybridization therefore takes on a much less metaphorical meaning and concerns entirely the ways man and computer can have “naturalizing” relationships.
8 With this concept as starting point, Couchot studies the concrete and artistic applications of the externalization of cognitive processes, first in computers, then in “artificial beings” (Couchot, 2013, p. 18). To what extent can our cognitive processes be modelled and encrypted into computer language in order to produce an aesthetic experience? Which forms can this externalization take? According to which criteria? To answer this, we need to distinguish between two levels of sophistication: that of automation and that of empowerment. In the first case, human cognitive processes are aided, guided by a machine that can model and process them faster and more accurately. In the second case—empowerment—digital technologies create the emergence of a self-organizing behavior. Then the computer can surprise us. This phenomenon is to be linked to the concept of autopoiesis, which describes the capacity of living systems to self-produce and self-regenerate. This leads to questioning the technical externalization of our sensations, emotions, memory, thoughts, and so on.
9 In a first categorization of artificial beings in 2013, Couchot makes a distinction between robots and artificial humans (Couchot, 2013). In this essay, it is worth stressing the importance given to the notion of empathy which conditions the possibility of interaction with artificial beings; artificial humans and robots are to be discriminated by their different technaesthetic experience. But since his second typology in 2018, he extends this theoretical system to its full potential (Couchot, 2018). If the level of automation is useful to categorize machines according to their complexity, Couchot agrees with Simondon’s analysis, considering automation as a “fairly low degree of technical prowess” (Simondon, 1969, p. 11). Eventually, the two most important criteria for Couchot are materiality and autonomy.
10 Materiality, both as medium and container, characterizes the “distinct modes of existence” (Couchot, 2018) that make it possible to discriminate between the different artificial humans: virtual humans and robots. And in between we can mention cyborgs, biological beings with have grafted mechanical parts. As far as virtual humans are concerned, their “mode of existence consists in representing a human, in virtual form, within a virtual space” (p. 24), whereas robots “possess bodies we can see and touch, and they act on real objects” (p. 25). Even though images have some form of materiality, Couchot proposes to distinguish virtual humans from robots according to their environment of interaction. This distinction allows us to analyze the level of complexity of the autonomy of these two categories. For Couchot, materiality is above all a mode of existence. We think, on the contrary, that this characteristic can also be considered as a variable level. Indeed a level of materiality invites to imagine virtual humans with a hint of materiality (for example, force feedback) or robots with a touch of virtuality (except for purely mechanical robots, all physical processes are also computer-simulated).
- 7 The category “autonomous actor” may create confusion with the more popular category—in the comput (...)
11 In addition to materiality, Couchot suggests to observe the level of autonomy and determines two extremes: the puppet avatar (zero degree of autonomy) and the autonomous actor7 (maximum degree of autonomy). Again, the system does not intend to restrict one qualification or the other, but rather invites us to think in terms of tendencies towards one extreme or the other. The puppet avatar, devoid of any autonomy, is but a projected image, realistic or imaginary, but we can nevertheless imagine avatars with some autonomy. To the opposite, autonomous actors “have a representation of the world around them (objects, other computer-generated actors, avatars representing humans) and […] are likely to communicate with them” (p. 24). This classification system also invites to imagine these virtual humans with a variable level of autonomy, for example guided by an external operator. Here autonomy is opposed to human intervention.
- 8 This matrix does not appear in the 2013 and 2018 articles by Couchot. This is a new and original (...)
12 Finally, Couchot notices the increasing presence of robots in everyday life. This observation leads him to call “almost-alive” a category of artificial beings capable of empathizing with us, with which we could live on a daily basis. Yet Couchot remains quite doubtful as to their effective production. We can summarize Couchot’s system of artificial humans in a matrix8 with two axes corresponding to the degree of materiality and the degree of autonomy. We present a visual representation of this matrix in the following diagram (fig. 1):
Figure 1
Matrix inspired from Couchot’s classification distinguishing the levels of materiality and autonomy of artificial humans in his publications of 2013 and 2018.
Diagram by Rémy Sohier.
13 The matrix allows us to see the possibilities offered by varying some parameters, in order to obtain artificial hybrids with various degrees of materiality and autonomy. We can also notice in the actual system a relatively voluntary absence of intermediate categories of materiality. It could be argued that “almost-alive” beings are not mainly characterized by their autonomy, but rather by the combination of virtual persistence and the ability to interact with material elements around us. Finally, this typology also experiments with the flexibility of avatarization, which is a kind of hybridization between a natural human and a virtual human, according to varying degrees of materiality and autonomy. In order to explore the rich possibilities opened by such a matrix, we propose a research-creation dynamics to test the plasticity of these levels of materiality and autonomy.
14 Couchot’s typology is of artistic interest to us because it has a lot of variable parameters. As stated earlier, artificial humans do not belong strictly to one category or another, but are rather defined according to varying attributes. The aim of the research-creation described below is to experiment the artistic plasticity of this system, pinpoint its strengths and its limits, in order to nurture a thought about multiple avatarizations. How can Couchot’s system help us understand situations where several people control the same avatar, or on the contrary, where a single person manages several avatars? We chose to call these cases “co-avatarizations.”
Figure 2
The Wizard of Oz virtual human controlled by a participant, during the exhibition of the 3AI project: Immersive Improvisation in Interactive Arts, at the National Tsing Hua University (NTHU) in Taiwan, in 2020.
Photograph by Rémy Sohier.
15 According to Couchot, hybridization with technologies for artistic purposes, in particular virtual reality, requires interactivity. Our first experiment to test this thesis, called the Wizard of Oz, is a sort of technological prank, presented in 2020 at Tsing Hua University in Taiwan (fig. 2). The experiment is inspired by the computing methodology of the same name (Green & Wei-Haas, 1985), which consists in using a human pretending to be a machine to simulate complex interface and interactivity processes that could not be modelled in advance (Alevêque, 2019). Our approach takes the form of a video projection of a virtual human being that can be animated at will. This setup relies mainly on a technical operator, hidden from sight, who can discreetly control the virtual human (fig. 3). The operator can move all parts of the virtual body (arms, head, appearance, position on screen…). However, there is no technological tool for capturing the participant’s movements (such as a camera, Kinect captor or motion capture device). Nevertheless, from the participants’ point of view, as soon as they come near the projection, the virtual human reproduces their movements—thanks to the malicious hidden operator. Different interactions follow, between the participant and the virtual human, perceived as the avatar of the participant himself. Sometimes, the virtual human offers other gestures, to invite the participant to engage in other types of interaction. Therefore, the operator reproduces the machine automatism and autonomy of an empowered avatar (or agent). The experiment always ends with the participant laughing when they discover the deception afterwards.
Figure 3
The Wizard of Oz control device managed by the hidden technical operator at the 3AI exhibition.
Photograph by Rémy Sohier.
16 As aforementioned, one of the criteria defining avatarization would be interactivity. But does the interaction between participant and avatar have to be embedded in the code? Can someone have the illusion of interacting with an avatar, and experience a similar aesthetic sensation as if they really controlled the avatar? The Wizard of Oz installation is a means to play live with the level of autonomy, in order to move from a technaesthetic experience of avatarization to a technaesthetic experience of interaction with an avatar or autonomous actor (at least appearing to be). The level of autonomy is assessed by the operator’s sensitivity and empathy for the participant’s background. It is the we-subject of the technaesthetic experience that makes it possible. In other words, the cultural codes of interaction with avatars. We can list some of these principles: video projection of an avatar, waiting for a participant to react, synchronization of movements between participant and avatar. These codes contribute to the illusion of interactivity and the deception is noticed only afterwards. At the time of the experiment, it is not perceived as a plastic parameter, contrary to illusions of perception in virtual reality (Guez, 2015). The attention to the participant’s body, usually taken in by a motion capture device, is relegated to a human operator simulating the process. Therefore, the artist plays with the autonomy parameter and observes the participant’s behavior, then comes up with a kind of improvised choreography. If we copy Couchot’s terms “externalization of cognitive processes,” we could speak, with a touch of irony, of the internalization of artificial processes. But it is above all an exercise in empathy for the artist-author of this setup.
17 Once the prank is revealed, the category of non-autonomous avatar emerges. The artistic gesture takes form in this very discovery and in this sudden change of autonomy level: the virtual partner, so far perceived as an autonomous actor, becomes a virtual puppet.
18 Unlike the autonomy level, varying the level of materiality complicates the implementation of an interactive experience, because materiality cannot be developed. Matter is not to be produced or transformed at will with as much freedom as the autonomy of an avatar. However, bodily involvement with the interfaces can be modulated. How can we test these variations of materiality through various degrees of bodily involvement?
19 In answer we designed a setup called Game Jockey enabling a live performance during which an external operator can change the rules of the game in real time. Here the Game Jockey refers as much to the device as to the external operator himself, such as a disc jockey (DJ). The core concept of the Game Jockey is to go from one playful experience to another, through a technical operator external to the players, and that this succession creates an aesthetic experience. This power to change the rules of the game enables us to simulate rule systems that are more sophisticated than in the Wizard of Oz experiment. Indeed, the operator is not hidden and his interaction with the participants is essential to the experiment. We have the players on one side, the Game Jockey on the other and the game between the two, so that a three-way relationship is created around the shared experience of controlling avatars. Moreover, there is a group of players, who can compete or cooperate and can also control a same video game avatar, namely a hero in an adventure game. The whole system is thought for a co-avatarization that can vary over time. The device can simulate game situations and quickly involve players in the creative process (Sohier, 2019).
Figure 4
The Starship Game Jockey with its six floor buttons, during a presentation at the Cosmodule, in Lyon, November 24, 2018.
Photograph by Rémy Sohier.
20 To be more explicit, let us present the example of the Starship Game Jockey (fig. 4) designed to test the level of materiality. The Starship device consists of a video projection of a game; a technical operator (the Game Jockey) beside the screen, facing the players; six floor buttons; several players interacting with the buttons and communicating visually with the Game Jockey. Thanks to the buttons, the players control different parts of a spaceship and move from one game situation to another.
21 The Game Jockey makes sure that the rules slowly evolve towards situations with varying difficulties, different visual and audio representations, all to the rhythm of a rapid succession of music pieces. The players control the spaceship, but so can the Game Jockey. They are co-acting in a virtual malleable universe where humans communicate through play. This leads to situations where players need to observe quietly, then act rapidly, or coordinate their actions spatially. Their physical involvement evolves over time as well as in space.
22 With the Wizard of Oz, the idea as to vary the autonomy of the avatar in order to bring out different postures in the participants, with Starship, in order to vary the level of materiality, the Game Jockey modifies the bodily implications, the efforts required, but also the spatial arrangement of the cooperating players. Modular buttons on the floor allow this reprogramming during the game. However, the experiment does not allow to carry out an in-depth exploration of changes in spatial disposition of the controls, which would lead to different materialities. To conclude, the degree of materiality seems to be more complex to change than the level of autonomy. It would be interesting to consider replacing it by a criterion of bodily involvement, but the difference between avatars and robots would be obsolete. We believe that the level of materiality and bodily implication can easily work in a complementary way.
Figure 5
The setup for Fleeing War with two diptych screens and a joystick to control both characters at the same time, exhibited at the Creative Gaming Festival in Hamburg, Germany, 2018.
Photograph by Rémy Sohier.
- 9 Fleeing War was co-produced by Vaclav-Havel library in Paris and La Cimade association, an organi (...)
- 10 The “game feel” refers to a global approach to the perception of motions controlled by interactiv (...)
- 11 Pseudo-haptics is a sensory illusion based on a discrepancy between our anticipation of the movem (...)
23 As we mentioned earlier, avatarization is often limited to a dual relationship between a participant and his or her avatar. In our interpretation of Couchot’s system, we imagined the possibility of an external operator acting with the artificial intelligence of the virtual human. But what would happen if the same person controlled several avatars? Could the complexity of avatarization be a tool for plastic realizations in the same way as materiality or autonomy? In order to offer a partial response to this interrogation, we designed Fleeing War, a video game presented in 20159 (fig. 5). We developed two versions of the game. In the first version, this creation proposes a narrative and political game experience, in order to question our perception of migration trajectories in France (Sohier, 2016). The collective Alineaire developed the game trying to emphasize the complexity of our representation of migration. To do so, the game takes place on two screens (in a diptych arrangement) where two characters—one on each screen—can be controlled simultaneously. A joystick allows the player to command exactly the same actions for both characters on their respective screens. The game tells a story about crossing borders and gradually some minor variations appear, making one of the two characters slow down, shake, fall over or disappear. These variations exploit game-feel techniques (Swink, 2008)10, pseudo-haptics (Lecuyer, 2000)11 or interactive narration. The aim of these variations is twofold: first a narrative effect to pinpoint similar migration trajectories, but ending in tragedy for one of the characters; secondly, a cognitive means to question our perception of avatar control in the context of a double avatarization.
24 In Fleeing War, the action gets more and more complex, as it requires the player to focus on two characters at the same time. Unfortunately, the experience was rather flawed in this first version. Indeed, feedback from players stressed how difficult it was to concentrate both on the characters and on the story. The controls were too complex and did not allow for a smooth hybridization with the technological device.
25 After this first experiment, a second version of the game was conceived, in which the characters were no longer controlled at the same time, but successively. In this new game, the story was segmented in chapters that had to be played once with the character on the left, once with the character on the right. In each chapter, small variations of movement appeared, enabling the players to grasp them more easily. So in the first version of the game, the aim was to experiment with the possibility of avatarization with several characters at the same time, with the same interactivity, which does not seem to be possible on the cognitive level; whereas in the second version, there is enough time to empathize with each character separately, with a different type of interactivity each time. This approach places an emphasis on the technaesthetic experience building up with each avatarization experience: each character’s personal history becomes new knowledge in the background of the controls, allowing their problematization in a similar situation, with some variations. This approach allows the development of schemas that are successively taken down by the interactive narration. These subtle shifts in avatarization create a powerful plastic parameter for the benefit of a fun interactive narration.
26 The avatarization in Fleeing War is very different from that proposed in the Wizard of Oz. However, it is not obvious to identify it according to the levels of materiality and autonomy proposed in the matrix of artificial humans. But the succession of juxtaposed avatarizations is indeed a variation of parameters. We have to mention now that the character on the left has all the characteristics of a puppet avatar, sufficient for the exploration of a virtual universe. Yet the character on the right has a touch more autonomy: he can avoid obstacles, express that he is tired, or resist our will to control him. While this form of autonomy may be perceived as important from a player’s point of view, it remains quite simple compared to the emergence of a form of intelligence or to the externalization of empathy in an autonomous agent. Nevertheless, in a narrative context, with a comparative approach to avatarizations, this simple variation of autonomy with a puppet avatar is enough to be meaningful.
27 Couchot designed a classification system of virtual humans according to two main levels of variation, which we simplify as follows: the level of materiality and the level of autonomy. This approach is made possible by the externalization of cognitive processes in artificial beings, such as intelligence or empathy. These artificial beings can take the appearance of a computer-generated image or a robotic form. The level of materiality allows to imagine intermediate states, but we showed that applying this variation is more complex when changes in real-time occur.
28 From an artistic point of view, Couchot’s classification system of artificial humans harbors a certain plasticity procured by the levels of materiality and autonomy. In order to vary the autonomy throughout an interactive game experience, we designed experiments with an external technical operator who can substitute for artificial intelligence: the Wizard of Oz and the Starship Game Jockey. The choreographic performance in the Wizard of Oz and the three-way performance of the Game Jockey produce different simulations of interactions with a virtual human. These experiments offer a reflection on our inter-individual empathy via avatarization technological tools and a demonstration of the artistic plasticity offered by the variations of autonomy.
29 We also experimented with the level of materiality in live performances, but this does not seem to be possible in terms of materiality of interfaces, because it is quite difficult to make matter appear or to transform it. However, we can easily modulate our bodily involvement with the interfaces. Then it is possible to imagine inter-individual bodily variations involving interfaces, as would a digital ballet. This level of bodily involvement seems to us more interesting in terms of plasticity, but we should not reject the importance of the level of materiality conceptually.
30 Finally, we tried to experiment with multiple avatarizations, thinking that the complexity of avatarization could also become a new tool for artistic expression. The experiment was not very successful, but it emphasized the importance of the technaesthetic experience that Couchot called for in technological hybridization. Indeed, there is an experience of avatarization that can be compared to previous avatarizations, taken in a narrative context. If the gap between a puppet avatar and an autonomous actor is important, simple interactions with their environment can instill an additional and meaningful empowerment of the avatar. This last experiment highlights the substantial plastic potential of the system.