Navigation – Plan du site

AccueilNuméros10DossierAI for Facial Autobiographical Me...

Dossier

AI for Facial Autobiographical Memory Recovering and Transmission. A Study for a Mobile and Web Application

AI per il recupero e la trasmissione della memoria autobiografica facciale. Uno studio per un’applicazione mobile e web
Elsa Soro, Silvia Barbotto, Remo Gramigna, Antonio Dante Santangelo, Cristina Voto et Massimo Leone

Résumés

Drawing on research conducted within the ERC Consolidator Project FACETS, this presentation aims at describing the state of art of solutions currently offered by AI for coping with loss of memory and identity in the pandemic scenario and at discussing advantages and disadvantages of these digital tools.
On the one side, the presentation will focus on how the interpersonal physical distance imposed because of the risk of contagion might be dealt with by applications based on Artificial Intelligence purposed to mitigate the loss of memory in elderly people by building new bridges between them and the generation of digital natives. On the other side, the paper will concentrate on obstacles that hamper the adoption of Artificial Intelligence across the generations. Lack in digital literacy as well as inappropriate interaction design as obstacles to effective cooperation between humans and artificial intelligence will be particularly stressed.
Finally, the presentation aims at presenting a concept study for a mobile and web application for autobiographical memory recovering, digitalization, preservation, recreation and transmission. The app is intended to bear on the visage as a memory and identity catalyst so as to enhance the neurophysiological functions of users, training their memory, and improving their ability to communicate.

Haut de page

Notes de la rédaction

Kutumia teknologia akili bandia (AI) kurudisha na kusambaza kumbukumbu. Utafili kuhusu kutumia simu za mkononi na mtandao

Kutokana na utafiti uliofanywa na mradi wa ERC iliyo ndani ya FACETS, katika habari hizi, tunaelezea maandalizi yanayopatikana kupitia AI kushughulikia hali za kupoteza kumbukumbu, kutambua ukubwa wa janga na kuzungumzia faida na upungufu wa teknologia hizi za kidijitali.

Kwa upande mmoja, habari hizi zinakazia jinsi ya kutumia teknologia zinazotegemea teknologia akili bandia (AI) kushughulikia hali kati ya wazee wazee wanapodumisha umbali ili kuzuia maambukizi na kizazi cha wanadijitali. Kwa upand mwingine, habari hizi zitakazia fikira hali zinazozuia teknologia ya akili bandia (AI) kuanza kutumiaka na vizazi vyote. Tutazungumzia jinsi ukosefu wa uzoevu wa kutumia vifaa vya kidijitali kutia ndani vikwazo vya ukosefu wa muundo unaofaa wa mwingiliano unavyozuia wanadamu kutumia teknologia ya akili bandia (AI) itakaziwa. Ukosefu wa uzoevu wa kidijitali na pia miundo isofaa ni vizuizi vikubwa vya ushirikiano wa wanadamu na akili bandia (AI) itakaziwa.

Hatimaye, habari hizi zitaandaa utafiti mdogo unaozungumzia jinsi simu za mkono na mtandau zinaweza kutumiwa kurudisha kumbukumbu, matumizi ya kidigitali, kuhifadhi, burudani na usambazaji. Program hii inakusudiwa kuwa chanzo kutafuta njia ya kuwasaidia watumizi waweze kuzoeza kumbumbuku lao, kuongoza ubongo wao na kuboresha njia ya kuwasiliana.

utambulisho kutumia uso, kawasifu, semiotiki program za kutumia uso

Texte intégral

This article results from a project that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement No 819649 - FACETS).

Background

1According to recent data provided by Eurostat (2019) in the European Union, older persons (aged 65 or over) had a 20.3 % share, with a significant increase compared with 10 years earlier). Beyond the adagio “More years, better life”, the challenges posed by ageing population target in particular social protection systems and health and long-term care since the yearly increases in life expectancy often mean a higher number of years with disability and chronic diseases. Notably increasing longevity, by entailing significant changes in family structures, should be considered a process that affects all ages. 

2The current COVID-19 pandemic has emphasized the needs and vulnerabilities that older persons have been suffering. Beyond the drama of the high mortality rates in older people registered during the pandemic at global level, the measures of social distancing have dramatically impacted the already precarious situation of older adults in assisted living residences, who have seen the face to face encounters with family members reduced for safety reasons.

3If the rest of population has tried to cope with the absence of face to face contacts, distance through a massive digitalization, the elderly have been further excluded by the digital divide and confined at home could hardly participate in the forced world experiment of face digitalization. 

4Undoubtedly, the face is one of the most important identity signifiers and the most represented object in human history. New technologies for face representation have satisfied an emerging sense of individuality, yet they have also promoted it. Nonetheless the pandemic stressed that face-based social networks have been mostly designed by young people for young people and they arguably failed in connecting people who are distant in time, such as the young generation with the older ones. In particular, they seem to be oblivious of memories that are enshrined in pre-digital photographs, in old albums, belonging to old adults who now live alone in assisted living residences, far from their families. Ultimately, the loss of autobiographical memory and identity in elderly people inevitably leads to an impoverishment of intangible heritage preservation and transmission, which affects the society as a whole.

5Yet, there is an increasing interest in expanding the role of new technologies in connecting generations through co-learning processes that often incorporate playful and co-design tools. In particular, Artificial Intelligence might help to cope with such autobiographical loss, by offering the possibility of creating new visual facial autobiographies that might lead to meaningful reconnaissance narratives about the self.

6In what follows, the paper stems from an overview of vast landscape of existing autobiographical and facial apps. The methodology used in the selection of the existing applications has been based on a benchmark of the first entries from a desk search on both Apple store and Google Play

7On the basis of the gaps identified within the scope – to allow the elderly persons interact at a distance with their loved ones – the concept of a new application will be drafted and its main functionalities will be illustrated.

A bird’s eye view of apps with reference autobiography

8The idea of autobiography has a long pedigree and it is generally linked to the written medium. Nonetheless, in recent times, the increasing need of digitalizing the self-story, leads the market to release several new apps of visual storytelling that tell the biography of a users through images.

9In the vast territory of visual storytelling, in the sample period December 1st – January 1st 2020, we selected a cluster of applications sorted both in the Apple Store and in the Google Store, by using the keywords “face” e “autobiography”.

10Some of the selected apps focus of personal memory and by so doing capitalize on this concept. Such apps are an extension of memory. Indeed, these apps seek to enhance and empower personal memory by incorporating the idea of the digital archive in a mobile phone. Memelife and Momento are two illustrations of this category of face apps. Also, digital family albums, life memories / special moments, and genealogy apps fall into this basket. In a nutshell, all these types of apps share the feature of capitalizing on the human need to remember. Some applications are bent towards family history and trace back the root of a family through a visual genealogy – like Ancestry – or guide the user to make a family tree. Quick Family Tree and MyHeritage illustrate well this trend.

11Moreover, quite rich and heterogenous is the panorama of apps that focus on collage of family pictures as well as collecting memories of special moments in life, from birthday parties to weddings.

12However, from the analysis of the current app concepts, we found a gap to be filled : the notion of face as a visual story. In that regard, this paper presents a concept of facial autobiographical app in order to help the recovering, digitalization, preservation, recreation and transmission of the self memory

The landscape of facial apps

13As mentioned above, the face is the symbol par excellence of the human being and a powerful cultural device. However, the face is not so easy to frame and any system that allows the design, representation, and communication of the face tells us about the society that produced it (Gates 2011, Belting 2013, Leone 2019). This is because the face is strongly connected to identity but, at the same time, it is always ready to change, and, during our lives, we witness a continuous transformation of the face we have into the face we make and share socially. This inter-faced aspect underlies that this device at the threshold between biology and culture is capable of connecting and separating not only the interiority and the exteriority of a single person, but also the individual from the society. 

14Since the 1960s, computer vision has been interested in the face and specifically in the possibility of its recognition (Wechsler H., Jonathon Phillips P., Bruce V., Fogelman Soulié F., Huang T.S. 1998 ; Cipolla, Battiato, Farinella 2010). It should come as no surprise that sixty years later, face recognition is still the epicenter of face research. There are many mobile applications that use face recognition systems for different purposes such as surveillance, work, dating, or entertainment. In the following, we offer a quick overview of the most common ones, according to a search in both Apple and Google stores.

15Firstly, it is worthy mentioning those applications that use the face as an eye-tracking device are being widely appreciated in medical and engineering fields (i.e. : Eye Tracker app or Umoove app).

16Secondly, our daily experience confronts us with a perpetual mobile of faces that we use and share thanks to the mimicry that adapt the social roles that we inhabit to the image of self that we want to share. In this regard, another important aspect of facial mobile apps is the morphing as a transformation of the face made through different visual strategies like cosmetic, prediction, cross-identity, cross-gender, and, cross-interspecies among others (i.e. Face Changer app or Face Warp app).

17Furthermore, the face is also a powerful communication device able to put in contact with our interiority with our exteriority and to map our health. This is why a third group of facial apps can be conceived by those apps that use the face as an image for predicting and monitoring wellness and health of the users (i.e. Face Exercise app or FaceYoga - Facial Health & Fitness. With respect to the apps we have mapped, one peculiar aspect of the face seems to be missing : the capability of making one's visual biography visible and, in this way, making one aware of one's visual identity.

18From this first reconnaissance, there seems to be a blank space left by the possibility of conceiving the face as a device for the reconstruction of the visual identity of the self : the face understood as an autobiographical and memorabilia device. Throughout our lives, we collect an enormous amount of facial representation that also permits us to efficiently and correctly move in our societies. This feature has a remote history which effects are visible to everyone in that common visual phenomenon known as pareidolia. Pareidolia refers to a peculiar perception, whereby we tend to attribute shapes, often of faces, to natural or random conformations of objects. It is therefore a particular case of "apophenia", which is the attempt to establish meaningful connections on a perceptual level in the attempt to order an otherwise senseless panorama. This visual phenomenon has a biological evolution linked to the recognition of faces as tool for the surviving in hostile environments and is capable of occurring both in newborns (Reid et al. 2017, Buiatti et al. 2019) both in persons who suffered a visual lost, as the Charles Bonnet syndrome testifies. 

19For all these reasons, we believe that understanding the face as a device usable through mobile apps can be a powerful and useful tool for recovering memories not only in a pathological perspective but also within a cure-caring purpose for elders. From a survey of mobile apps dedicated to memory retrieval by the elderly (i.e. Luminosity Brain Training or Memorando Brain Training), it would appear that they all aim at usability for elders exclusively in the design of the interface and not in the narrative of use. Moreover, the proposed goals are always internal to the storytelling and do not aim at the creation of any extra/appropriate textuality by the user and his family.

Drafting a new app : methodology, scopes and functionalities

20On the basis of the research carried out in the preliminary phase of our research, we decided to use the methods of structural semiotics and generative narrative theory to design an app that would tell the story of elderly people and encourage them to interact with texts about themselves and their identity. The aim of the app is to allow the elderly persons using it to interact at a distance with their loved ones, creating a social network that enables them to construct and participate in discussions about themselves and their history, which can be shared and, if necessary, taken up again in their presence, when it is possible to meet. These discussions can be based either on memory, reconstructing the elderly persons' past, or on imagination, allowing them to reflect on how their life could have been if it had been different. In both cases, users will be able to reflect on the identity of the individual using the app, on how this identity has actually been determined in real life, but also by difference, i.e. how it could have been configured if the course of historical events or the choices made had defined it in another way. In fact, one of the most important theoretical approaches of modern structural semiotics, from which many of the reflections on which our work is based derive, is centered precisely on the recognition of the identity of things, persons and events, according to the similarities that can be inferred between them, because of the differences that distinguish them from other things, other persons and other events that can be linked to them (Saussure, 1916). 

21The operating principle of the app is very simple : it would aim to build multimedia files generated from the contents of a database, populated partly by the elderly persons using it and partly by a duly trained Artificial Intelligence algorithm. Both should upload images, videos, music, written or spoken texts, meaningful to tell the users' story : the latter selecting them from their memories, perhaps drawing on their digital devices, if they have them, or scanning or translating into bits what, of analogue, they carry with them from their past, such as old letters, postcards, printed photographs, music recorded on vinyl records, etc. ; the AI, instead, should go on the Internet, fishing from public repositories various forms of textuality that refer to the historical periods in which the elderly users of the app lived. In this way, the database would be populated precisely from a difference endowed with meaning, between 'subjective' content, i.e. filtered from the point of view of the persons who want to tell their personal stories, and 'objective' content, external, linked to history with a capital 'S'.

22Another programme would then take care of catching and editing together, putting them side by side and linking them, some of these same contents, in a way that would be able to stimulate reflection and discussion. For example, it could produce a photo album, taking a series of images representing moments in which the elderly persons have been happy throughout their lives, perhaps focusing on their face and how their smile has not changed over the years. Or it could create a video with an oral commentary by the elderly persons about their happy memories, accompanied by these photos. Or it could combine images of important historical figures with those of the users' faces, as they were at the time when those historical figures were famous. It can create a gallery of the most representative design objects and clothes of the time when the users were young, perhaps by juxtaposing photos from old fashion magazines with those in which they were dressed or styled in the same way. And so on.

23For all this to be possible, the mechanism on which the app would be built is that of classifying the contents of the database by means of tags conceived on the basis of semiotic theories on narrative (in particular, those of Greimas, 1970 and 1983). These tags, used either manually by the user, or automatically by the algorithms, would make it possible to retrieve and process these contents in a way that is not explicitly narrative - the app would not tell stories as a human being would - but proto-narrative, i.e. based on the general rules of narrativity (Lorusso, Paolucci and Violi, eds, 2012). This would give rise to multimedia products that clearly reveal the personal stories behind them, stimulating the users of such contents to tell their stories and the audience of their loved ones to ask them to tell.

24Since narrativity, in its most general sense, is nothing more than a way of linking, according to some logic, the states of things and the events that modify them, thus assigning a specific meaning to these changes - whether they occur, or are configured as a mere eventuality (Ferraro, 2015, pp. 15-30) -, the tags on which the app would be based would be aimed, precisely, to facilitate a classification of the contents uploaded by users and its algorithms in this perspective. They would be divided into tags designed to identify the theme or topic of such contents and others designed to identify the rheme or focus (Van Dijk, 1977).

25Some examples of 'topicising' tags would be :

26- chronological tag : date of the period when that content was made ;

27- descriptive tags : names of people, family ties, places, events referred to within that content.

28Some examples of 'focusing' tags would be :

29- connective tag : what certain contents can be related to (terrorism, economic boom, wars, protests) ;

30- values tag : what the contents referred to have represented for the elderly users, in terms of emotions or concepts that can produce emotions : e.g. happiness, freedom, responsibility, fear, etc. ; and

31- actantial tag : the role that the things, people or events represented within certain content have played in the users' life history (object of desire, helper, mentor, enemy, opponent) ;

32In doing so, the app could create meaningful links between the tagged contents, grouping them together, for example, by related types : all the contents that refer to the same year, all those related to the economic boom, those in which the users are portrayed with their wife or husband, those that represent them in the moments when they felt free, those that talk about their marriage or that talk about marriages in the same period in which they got married, those that represent what they wanted in life, those that represent who helped them, etc. Of course, the app could also create temporal sequences of items classified with the same tags. For example, it could show all images of the same persons over time, or of their family at different times and in different places. But more importantly, it could cross-reference items with different tags, building collections of content in which the older persons have felt happy or empowered over the years, or of the different protests they have participated in, or of the different holiday destinations they have gone to, and so on.

33To make the app's AI algorithms even more efficient in creating multimedia montages that tell the users' story, they would be trained by the users themselves, learning from the way they classify content about their lives. In this regard, in a special section of the same app, elderly people would also be asked to undergo an interview, in which they would be asked some questions aimed at framing events, characters, situations and thoughts of their lives, in a proto-narrative way, according to the logic of the tags mentioned above.

34Moreover, in this section, users would be asked some questions necessary to create another type of tag, the virtualising tags. They would be asked what job they would have liked to do, among the ones they did not do, where they would have liked to live, among the places they did not live, what famous person they would have liked to meet, what clothes they would have liked to own, and so on. In this way, the algorithms themselves could be prepared to create multimedia montages based on a fundamental mechanism of narrative, i.e. that of the "what if", giving rise to very interesting content to comment on, both for elderly users and for their loved ones, especially in that ludic perspective that we talked about in the part of our work dedicated to the survey of face apps and to the most effective ways of involving elderly people in activities that lead them to think about themselves and their history.

35All these original features, which of course would need further study in order to be implemented in the best possible way, could also be complemented by some of the features that already exist in the world of apps, from morphing to the use of filters. In both these cases, one could insist on the fundamental narrative question of 'what if', for instance by proposing to the elderly people to look at what their face or figure might have looked like, in different historical moments, if they had adopted different lifestyles (the app algorithms should be able to represent users as hippies, sportsmen or sportswomen, intellectuals, etc.), if they had done different jobs (managers, factory workers, nurses, tennis instructors), if they had dressed according to certain fashions, and so on.

Final remarks : memory and what follows

36Memory, as a semiotic instance in a continuous constructive process, assumes an interactive value when we make it part of our lingüístic and corporal narrative. Working with and on autobiography, means doing it with and on memory, that implies re-dimensioning one's own identity in synchronic and diachronic terms, increasing the attention on the present as a co-constructive momentum of a shaping past. It means, however, not only to establish a dialogical dimension with the self, but also and above all to historicise the interaction with otherness : “face memorability is in fact a critical factor that dictates many of our social interactions ; even if a face seems friendly or attractive, if it is easily forgotten, then it is meaningless to us”. (Khosla, Bainbridge, Torralba, Oliva 2013 : 3200)

37We have seen that the use of app features games could cover all aspects of thinking, from spatial processing to reasoning. Thus the elderly can have a great user experience using the interfaces proposed : fresh brain workouts daily help to keep the mental abilities in check and the seniors could find it very handy as they need regular challenges and they could see benefits from improving problem-solving, flexibility, attention paying, speed, and memory. 

38We have also approached the role of expression, those that the elders have maintained all their life have built memory in the face, whether if we consider smiling, frowning, or the worry signs. The extra movement of those facial muscles actually etch an expression into the face : exercise can help to build memory consciousness. Increasing the knowledge of the different facial atrocities, expressive or internal, evident or latent, problematizing narrative elements solidified in time and questioning the degrees of memorability through the exercise of body language, enhances the work of memory. 

39Can we describe our face remembering exactly how it is without seeing it ? And face of otherness ? Which detail in common and which are the differences ? Can we remember how our face was when we were children ? The app we have proposed include memory, pictures and survival challenge : in remembering situations and activating intimate storytelling starting with own face perception stimulate the construction of the visual autobiography. 

40To elaborate this proposal it has been necessary to do an interdisciplinary, even a transdisciplinary job : “Today, it is fundamental that semiotic analysis turns to technology, not just in order to philosophize on it à la Bruno Latour (which is also important), but also in order to empirically study common everyday technical devices : smartphones, social networks, and especially apps.” (Leone, 2020 : 7)

41We know, however, that there are still many gap about digital studies and unknowns about memory studies, indeed "a proportion of memorability can be described by a combination of facial attributes (…) but there is still a large amount of unexplained variance to memorability reliable” (Baibrigde, Isola, Oliva 2013 : 1332) ; for this reason, and because of the desire to fill interstice or at least intertwine spaces of knowledge that are still distant, it seems relevant to continue working in this direction. 

Haut de page

Bibliographie

Bainbridge, W.A., Isola, P. and Oliva, A. (2013). The intrinsic memorability of face images. Journal of Experimental Psychology, 142(4), 1323–1334.

Belting H. (2013). Faces : eine Geschichte des Gesichts. Munich : C.H. Beck

Berle I. (2020). Face Recognition Technology. Compulsory Visibility and Its Impact on Privacy and the Confidentiality of Personal Identifiable Images. Berlin Heidelberg : Springer-Verlag

Cipolla R., Battiato S., Farinella G.M. (eds.) (2010). Computer Vision. Detection, Recognition and Reconstruction. Berlin Heidelberg : Springer-Verlag

Gates K. (2011). Our Biometric Future. Facial Recognition Technology and the Culture of Surveillance. New York : New York University Press

Greimas A. J. (1966). Sémantique structural. Larousse : Paris

Greimas A. J. (1970). Du sens, Editions du Seuil : Paris

Greimas, A. J. (1974). Les actants, les acteurs et les figures. In : Chabrol C. (eds.), Sémiotique narrative et textuelle (pp. 161-176). Larousse : Paris

Khosla, A., Bainbridge, W.A., Torralba, A., and Oliva, A. (2013). Modifying the memorability of face photographs. Paper presented at the International Conference on Computer Vision (ICCV), Sydney, Australia

Leone, M. (2020). Digital Cosmetics, Chinese Semiotic Studies, 16, 4, 551-580. https://doi.org/10.1515/css-2020-0030

Lorusso A, Paolucci C. and Violi P. (2012), Narratività : problemi, analisi, prospettive. Bologna : Bononia University Press

Saussure F. (1916). Cours de linguistique générale. Lausanne, Paris : Payo.

Van Dijk T. (1977). Text and context. Explorations in the semantics and pragmatics of discourse. London and New York : Longman

Walsh Matthews S., Danesi M. (2019) AI : A Semiotic Perspective, Chinese Semiotic Studies 15, 2, 199–216

https://doi.org/10.1515/css-2019-0013

Wechsler H., Jonathon Phillips P., Bruce V., Fogelman Soulié F., Huang T.S. (eds.) (1998). Face Recognition. From Theory to Applications. Berlin Heidelberg : Springer-Verlag

Haut de page

Pour citer cet article

Référence électronique

Elsa Soro, Silvia Barbotto, Remo Gramigna, Antonio Dante Santangelo, Cristina Voto et Massimo Leone, « AI for Facial Autobiographical Memory Recovering and Transmission. A Study for a Mobile and Web Application »Communication, technologies et développement [En ligne], 10 | 2021, mis en ligne le 20 mai 2021, consulté le 23 juillet 2024. URL : http://journals.openedition.org/ctd/6113 ; DOI : https://doi.org/10.4000/ctd.6113

Haut de page

Auteurs

Elsa Soro

FACETS - Face Aesthetics in Contemporary E-Technological Societies Team, University of Turin

Silvia Barbotto

FACETS - Face Aesthetics in Contemporary E-Technological Societies Team, University of Turin

Remo Gramigna

FACETS - Face Aesthetics in Contemporary E-Technological Societies Team, University of Turin

Antonio Dante Santangelo

FACETS - Face Aesthetics in Contemporary E-Technological Societies Team, University of Turin

Cristina Voto

FACETS - Face Aesthetics in Contemporary E-Technological Societies Team, University of Turin

Massimo Leone

FACETS - Face Aesthetics in Contemporary E-Technological Societies Team, University of Turin

Haut de page

Droits d’auteur

CC-BY-NC-ND-4.0

Le texte seul est utilisable sous licence CC BY-NC-ND 4.0. Les autres éléments (illustrations, fichiers annexes importés) sont « Tous droits réservés », sauf mention contraire.

Haut de page
Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search