Navigation – Plan du site

AccueilNuméros18-1ArticlesThe bodily challenges of augmente...

Articles

The bodily challenges of augmented vision applied to firefighting activity

Les enjeux corporels d’une vision augmentée appliquée à l’activité des sapeurs‑pompiers
Carole Baudin et Laura Maillard
Traduction de Christopher Hinton
Cet article est une traduction de :
Les enjeux corporels d’une vision augmentée appliquée à l’activité des sapeurs‑pompiers [fr]

Résumés

Les technologies de Réalité Augmentée (RA) sont en pleine explosion et commencent à être utilisées dans différents domaines d’activité. Toutefois, peu de travaux étudient leur impact sur les dimensions perçues par l’utilisateur en activité. A travers un projet de recherche visant à développer un dispositif de RA d’imagerie thermique pour les sapeurs-pompiers, nous avons étudié les dimensions perceptives à l’œuvre dans une activité médiatisée par la RA. A travers une méthodologie originale articulant l’approche anthropotechnologique et la cognition incarnée, nous avons étudié d’une part le rôle de l’imagerie thermique telle qu’elle est utilisée aujourd’hui dans les interventions incendie et les dynamiques sensorielles convoquées, avant de présenter les résultats issus de mises en situations réalistes réalisées avec un dispositif prototypique. Ces simulations nous ont permis d’observer de nouvelles formes d’agir avec le dispositif dans des environnements aux propriétés proches du réel et de poser des hypothèses sur la transformation des dynamiques perceptives et plus largement de l’engagement des corps dans l’action, générée par la vision augmentée.

Haut de page

Texte intégral

1. Introduction

1Digitalisation in the workplace is challenging ergonomics more than ever by calling into question not only the nature of jobs (Mériau, & Rousseau, 2017) and work-related values (Salini, Sarmiento, Goudeaux, & Poizat, 2018), but also the deeper changes it causes in our ways of thinking and acting in specific contexts (Baudin, & Nusshold, 2018). While this observation is becoming increasingly apparent in the literature, it would seem that there is still not enough analytical perspective to allow us to gain a detailed understanding of the dimensions of human activity brought into play via these new devices that are populating our professional activities, which some call “Digital Technical Objects” - DTOs, to emphasise the historicity of these techniques (Poizat, & Durand, 2017). Indeed, from an anthropological standpoint, and contrary to many discourses, the technological evolution we are experiencing cannot be considered as a revolution in the sense of a past discontinuity or threshold (Serres, 2013; Garçon, 2012; Levy, 2007). The technologies that have been appearing over the last ten years nevertheless involve “installation and deployment phases” (Valenduc, & Vendramin, 2019) that are new to the history of technology, particularly with regard to the place they hold in work structures: technologies of substitution or complementarity (Bobillier-Chaumon, 2017; Valenduc, 2017), transformation of our modes of intelligibility (Sadin, 2015) and corporality (Baudin, 2017a, b). So while there is no doubt that they will disrupt individual and collective professional practices, the lack of systematic empirical analysis does not yet allow us to understand their precise implications in the context of mediated activities. In particular, the Augmented Reality technologies that are gradually integrating certain professions (sales, maintenance, medicine) are only just beginning to be studied from the point of view of activity analysis, as the applications are still little known (Arnaldi, Guitton, & Moreau, 2018) or are still being tested.

2Unlike Virtual Reality (VR) which recreates a totally artificial environment, Augmented Reality (AR) “aims to enrich the perception and understanding of a real environment by adding relevant digital information. This information is more often than not visual, sometimes audible and more rarely haptic” (Moreau, & Normand, 2018, p. 55). By combining real and virtual data, these technologies aim to make it possible to “see the invisible or to re-see the visible” (ibid.) by integrating our perception of the environment through devices such as tablets, mobile phones or AR smart glasses. This integration does not therefore appear insignificant, especially when it becomes part of the body through the use of smart glasses. Such proxemics, which in some professions are intended to free the hands, presupposes a transformation of the body’s engagement in the activity concerned. By introducing new media that are as close as possible to the perceptual field, they bring into play one’s perceived relationship with the world, modes of perception and situational action.

  • 1 Project funded by the Hasler Foundation (“Cyber-Human Systems” thematic), run by Prof. Sabine Süsst (...)

3Our participation in the S.A.V.E. (Seamless Augmented Vision Equipment1) project, which aims to develop an AR thermal imaging device for firefighters, was an opportunity to study the impact of this technology on the perceptual dimensions of firefighting, during intervention simulations. After presenting a state-of-the-art of AR work, we set out the methodology and theoretical framework that we put in place to work on these aspects, and present the initial results obtained. The latter show the transformations generated by this technology regarding bodily engagement in the activity, thus allowing us to more broadly question the apprehension of perceptual dimensions in mediated activity, with the aim of laying the foundations for the reflection that has become necessary with the arrival of these DTOs in our daily and professional activities, and which some authors predict will generate a loss of meaning and a disembodiment of our practices (Le Breton, 2013; Haroche, 2008).

2. Perceptual phenomenon and Augmented Reality: the “seamless” challenge

4The aim of AR is to make “visible or palpable” information that our perceptual field does not acquire in situations where this information would be meaningful and structuring. Unlike Virtual Reality (VR), AR does not immerse a person in artificial reality, but instead aims to create a “mixed reality” by “inserting” information into a naturally perceived scene. From an application standpoint this creates two issues. The first is that of how the perception of “artificial” information (coming from the treatment of the real world) and “natural” information (coming from the perceived world) are combined. The second is the processing of mixed information (artificial and natural) which generates an unprecedented sensory stimulus for the “perceiving person”. These two points bring into play the phenomenon of fusion in the perceptual field of sensory stimuli of different types, which, from a technical standpoint is still difficult to achieve, and from a practical point of view raises questions about its possible effects. The S.A.V.E. project aimed to achieve this fusion, known as “seamlessness”, in its visual modality. This is a significant notion, because it reveals the objective of creating a specific mixed perception dynamic, at the frontier between natural and artificial perception. From both a theoretical and methodological standpoint, seamlessness raises the question of how to grasp the sensory (and more broadly, the perceptual) dimensions in the development and use of these technologies.

2.1. State of the art

5As a first step, our study of the theoretical corpus helped us to identify some of the problems posed by the application of AR devices to human visual perception – issues that have been studied over the last ten years in numerous articles in the field of computer science (Kruijff, Swan, & Feiner, 2010; Albarelli, Celentano, Cosmo, & Marchi, 2015; Baumeister, Ssin, & ElSayed, 2017). On the one hand a review of this literature allowed us to identify the different visual mechanisms at play in the application of AR systems, making it possible to clarify what would contribute to a “seamless” perception, and on the other hand, to analyse the approaches underlying these studies, and hence these results.

6Some of this research aims to identify the technical problems causing interference for human perception. From a visual standpoint for example, the phenomenon of matching the virtual image with the real image (Kruijff, Swan, & Feiner, 2010); the problem of distinguishing between objects in changing and cluttered environments (ibid. ); the problems relating to the absolute and relative positioning of objects in space (or reproduction of the notion of depth) (ibid.); the problems of occlusion of the real or virtual environment (caused by double vision over a long period) (ibid.), etc., are known and complex phenomena.

  • 2 “Visual fatigue is caused by repeated visual effort (for example, the repetition of major changes i (...)
  • 3 Visual discomfort is a visual problem immediately perceived by the observer through the integration (...)

7Some studies have made it possible to highlight the problems that users have in adapting to these technologies. While most of the research addresses known VR issues, a small number of studies have revealed problems more specific to AR: vergence-accommodation conflict due to a fixed 2D focal plane, which may cause headaches or visual fatigue2; visual discomfort3 related to monocular vision; physical fatigue due to head oscillation to compensate for the loss of visual field with some HUD devices (Claverie, & Léger, 2009).

8The utility of this research is that it establishes the perceptual interferences generated by AR and specifies the technical dimensions that remain “engineering bottlenecks” (Valenduc, & Vendramin, 2019) for this type of system. Most authors agree on the fact that the challenges of AR development relate to phenomena (such as visual focus or 3D vision - including depth vision) that are part of human perceptual intelligence (Moreau, & Normand, 2018).

9All of these studies regularly have to cope with the immaturity of technical systems that remain difficult to evaluate under real-life conditions. Unlike VR, applications of which have become more widespread in the last ten years or so, AR still suffers from a lack of experimentation in the professional field, because contrary to VR, the vocation of which is decision support in design and methods offices, AR is a “field” technology that aims to assist operators carrying out their activities “within production and construction units in the industrial world” (Arnaldi, Guitton, & Moreau, 2018, p. 39). But the equipment needed to be able to perform in such situations must be solid, reliable and intuitive, which is not yet the case. Very few studies evaluate these devices in the context of finalised activities.

10In so doing, the majority of the research is carried out under controlled conditions to assess the technical performance and limitations of AR by apprehending the perceived subject as a component of a binary device-subject system. The perceiving subject is here considered to be a sensor-motor system that experiences interference (AR) in its perception-action loop with the environment. This research is thus part of a positivist theoretical framework which considers perception to be a physiological, functional and cognitive capacity and which views the environment as objective data. These study situations are therefore often simplified and not very ecological. Moreover, Arnaldi, Guitton and Moreau (2018) note the theoretical and methodological difficulty of assessing such systems, particularly from the standpoint of understanding how they interact with users.

2.2. The SAVE project: achieving “seamlessness”

11Our participation in the S.A.V.E. project allowed us to lay the foundations for a “seamless” integration of augmented thermal imaging in the fire brigade, but also to question the conceptual and methodological tools enabling us to grasp the perceptual phenomena at play with this technology, and their role in a firefighting situation.

The project’s conceptual and methodological framework

12The theoretical and methodological approach adopted in this project is in line with the anthropotechnological current (Wisner, 1997; Geslin, 1999) developed over recent years in the User-Centred Product Design laboratory (laboratoire en Conception de Produits Centrée Utilisateurs) of the Haute Ecole Arc de Neuchâtel (Switzerland).

13Anthropotechnology emerged from activity-centred ergonomics and came about in response to the problems underlying technology transfer (Wisner, 1997). It is however separate from this current, as it clearly belongs to the currents of technical anthropology, cognitive anthropology and the anthropology of the senses. Its objective is to help improve working and living conditions for individuals and communities, by participating in the design of socio-technical solutions that take into account the ways people think and act in specific contexts (Geslin, 2017a). It articulates an understanding of activity at three levels: macroactivity (taking social, cultural, historical, economic and geographical dimensions into account); mesoactivity (taking into account the analysis of activity in its observable dimensions); microactivity (the study of situated and embodied practices of the construction of meaning and expression), in relation to ad hoc intervention methodologies, based on the principles of co-design with the actors concerned. This approach made it possible to lay the foundations of a study of firefighter activity mediated by an AR device, in its contextual, operational and perceptual components, and to put forward design suggestions for moving towards a seamless technology. Its more recent incursion into the consideration of sensitive, sensory and bodily dimensions is what led to the present reflection.

14From a methodological point of view, anthropotechnology draws on the principles of ergonomic intervention or design ergonomics (Daniellou, & Béguin, 2004; Béguin, 2004), based on the study of reference situations that make it possible to identify the determinants of an activity for a given community and context and, through extrapolation, to lay the foundations of the future system by analysing characteristic action situations (carried out in ecological or realistic conditions using figurative or operational simulation). This implementation makes it possible to establish the design criteria for a future system based on how future users think and act in context. As the problematic posed by the S.A.V.E. project concerned the sensory (and more broadly, perceptual) aspects, we adopted an analytical stance that focused on bodily engagement; this led us to propose a specific methodology (see part 1.3.3).

15In order to better grasp the perceptual transformations at play in activity mediated via AR, we based ourselves on an embodied cognition approach that draws on phenomenology research (Merleau-Ponty, 1945), ecological psychology (Gibson, 1979; Berthoz, 1997) and the theory of enaction (Varela, 1993). The postulate of embodied cognition is to recall that body and mind are not two distinct and separate entities; more specifically, it considers that cognition cannot be considered independently of the situations in which it originates - in this sense it is situated (situated cognition), is anchored in the body and emerges from its interactions (its embodiment) with the outside world - it is therefore ’embodied’ (embodied cognition). It is this incarnation of the organism that defines and limits the expression of cognition. However, from a methodological standpoint this theoretical approach has received little attention, except from course-of-action advocates (Theureau, 2004). In our opinion, from a conceptual point of view this research remains very costly and does not grasp the carnal dimension of the studied phenomena in all its complexity, because it focuses on a discursive and semantic interpretation. In this research, the logics and grammars specific to bodily intelligence seem to wither away.

16On the strength of projects whose focus on sensitive and bodily aspects has proven to be theoretically and methodologically fruitful (Baudin, 2016, 2017a, 2017b), we designed a methodology that integrates an approach which focuses on the body, in the sense of a corporeality (Bernard, 2001), which interweaves biological, clinical, psychological and cultural determinations that broaden the lived experience to the dimensions of an environment. In this perspective, we set up a methodology based on the study of gestures, including gesture in both its expressive and instrumental dimensions.

The user-centred design approach and contexts of use

17The empirical data studied in order to examine the perceptual transformations relating to the integration of AR in activity, are those that result from the co-design approach implemented within the S.A.V.E. project, the objective of which was to participate in the development of a technical demonstrator of AR smart glasses which make it possible to project a thermal image without affecting visual comfort and overall perceptual performance. This is what we referred to as “seamless”.

18The intended field of application being that of firefighters, the challenge was then to understand the visual and, more broadly, the perceptual dimensions at stake in the integration of technology in situations not yet (or rarely) explored in the literature and which have the characteristics of: i) being dynamic (in the sense of Hoc, 2004), ii) often critical, iii) and taking place in complex contexts/environments (we return to these in part 3).

19Using a design approach that focuses on users and contexts of use (Geslin, 2017a, b), we first carried out field studies (see Fig.1) to identify and then examine reference situations (the use of the thermal camera in fire intervention), which allowed us to understand the practices into which the technology had to be inserted and to identify the perceptual and technical aspects involved when integrating a “seamless” device into these situations. Secondly, we set up a co-construction dynamic with a panel of users via scenarios (based on typical action situations) of evolving AR prototypes and the gathering of data at every stage through subjective evaluation questionnaires, verbalisations resulting from the think-aloud protocol, and narratives and observations of gestures and routes. This helped us to draw up recommendations for the future system, and to observe the adaptation strategies implemented by users for working with the AR smart glasses.

Fig. 1. Diagram of the user-centred design methodology and contexts of use applied to S.A.V.E. project. 
Fig. 1. Schéma de la méthodologie de conception centrée utilisateurs et contextes d’utilisation appliquée au projet S.A.V.E.

Fig. 1. Diagram of the user-centred design methodology and contexts of use applied to S.A.V.E. project.  Fig. 1. Schéma de la méthodologie de conception centrée utilisateurs et contextes d’utilisation appliquée au projet S.A.V.E.

20The data on which the present reflection is based were therefore the result of this intervention, gathered from evolutive prototypes and not from a finished system. In addition, the objectives, with a transformative objective, presupposed a methodology not of diagnosis-intervention but of problem-solving-design. By studying and specifying the criteria for a “seamless” vision at each stage, we were nevertheless able to make hypotheses about the perceptual dynamics at work in the mediated activity we were studying.

2.3. Construction of the test protocol

21Scenarios were established through the synthetic modelling of reference situations (to be developed in part 3), allowing us to set up scenario simulations in accordance with experimental designs the variables of which related to i) the various environmental and spatial settings, ii) the thermal imaging tools used and iii) the characteristics of the participants, were systematically assessed in order to guarantee the representativeness of the situations studied. Moreover, the scenarios had to take into account the evolution of the prototype under construction; they were thus the compromise between a technological maturity and a methodological implementation seeking the internal validity of the study.

2.3.1. Evolutive prototype and simulation situation

Prototype V1

22We worked with two teams of image processing and embedded systems engineers to develop the AR system. It is basically composed of: i) a thermal image capture device (thermal camera), a projection and visualisation device (AR smart glasses with different projection options) and an image processing and management system. However, with this initial composition there are many available options that affect in different ways the vision of a thermal image on an AR scope.

23Based on the state of the art, we identified four AR configurations that could be tested to investigate the technical conditions for a “seamless” device. To this end we built a test bench, thus offering users different choices for capturing and projecting the thermal image (Fig.2): monocular or binocular projection, position and angle of the thermal cameras, single or dual-camera vision (to recreate a “stereoscopic” vision), two types of thermal camera with various different fields of view, resolutions and frame rates.

24Having made the choice on the test bench, the selected configuration was then loaded onto the person in question (cameras positioned on a helmet, smart glasses and a backpack to carry the image processing hardware) for the first usage scenarios (see section 4.2).

Fig. 2. “Prototype V1” allowing the choice of optimal configuration for each participant. 
Fig. 2. Prototype V1 permettant le choix de configuration optimale pour chaque participant

Fig. 2. “Prototype V1” allowing the choice of optimal configuration for each participant.  Fig. 2. Prototype V1 permettant le choix de configuration optimale pour chaque participant

(on the left, the test bench to test the different configurations; on the right, the different configurations offered) 
(à gauche, le banc d’essai pour tester les différentes configurations ; à droite, les différentes configurations proposées)

25The development of these test protocols also took into account the effect of the participants’ adaptation to the technique. We wanted to use an initial static test to accustom them to “seeing” with the prototype under conditions that were immersive but controlled, before placing them in more realistic conditions.

Prototype V2

  • 4 Fire mode: colorimetric scale from 0-650°, Person mode: 0-60°. Each mode has a “fixed” sub-mode (co (...)

26The V2 prototype is an improved version of the on-board device (Fig. 3). With the aid of a control box, the user could adjust the various display parameters of the enhanced thermal image to suit the requirements of each scenario. He could vary the positioning and size of the image, the colours of the thermal image (with two “fire modes” and two “person modes”)4, the image projection mode: monocular (an image projected onto the dominant eye) or binocular (the same image projected onto both eyes). The on-board system was designed to lighten the physical load during movement (freedom of movement, weight, autonomy, bulk) and to allow users to concentrate on the perceived dimensions of the environment and the situation.

Fig. 3. “Prototype V2” dynamically tested. 
Fig. 3. Prototype V2 testé en dynamique

Fig. 3. “Prototype V2” dynamically tested.  Fig. 3. Prototype V2 testé en dynamique

(left and centre: embedded system on helmet and remote control; on the right: the two configurations tested) 
(à gauche et centre : système embarqué sur casque et boitier de contrôle ; à droite les deux configurations testées)

27This phase allowed us to carry out tests in a realistic environment (the characteristics of which are very similar to the real environments encountered during firefighting situations) through figurative simulation (because we could not simulate a fire for reasons not only of risk, but also of technical constraints, as the on-board equipment had not survived real conditions).

2.3.2. Panel of participants

28Preliminary field studies and reference situations have shown the importance of firefighters’ (FF) experience in the management and use of thermal imaging during interventions. While experience is linked to different dimensions such as years of practice, or status (professional or volunteer) not involving the same level of engagement in the activity or the same responsibilities during interventions, we focused on experience with the thermal imaging camera currently used by most fire brigades, gained not only through frequency of use, but also through the level of training and instruction in the use of the camera. We made the distinction between an expert level (instructor and/or very frequent user of the thermal camera) and an intermediate level (user having received training, but with little use in live interventions). There were no novice level users (those having received no training and never having used a thermal camera) among the FFs because they had all received a minimum of training in order to be able to operate in the field. The novice level was therefore represented by two ergonomist-testers.

29We also took into account certain physical inter-individual variables such as the wearing of smart glasses, the dominant eye, visual acuity, etc. These parameters were however less decisive than experience data when characterising the impacts of situated AR.

30Finally, a panel of seven people carried out the two series of tests:

  • 3 professional firefighters with between 14 and 25 years’ experience; trained or instructors in the use of the thermal camera during interventions; frequent use (2 to 3 times a month).

  • 2 volunteer firefighters (with between 6 and 15 years’ experience), trained in the operational use of the thermal imaging camera; less frequent use (2 to 3 times a year).

  • 2 ergonomics experts (not firefighters), untrained.

31The choice of the two expert ergonomists is a methodological bias. Although they are indeed novices in the use of the thermal camera, they are nevertheless experts in the verbalisation and analysis of perceptual phenomena experienced in context. While the FFs could highlight device issues relating to the performances expected in an operational situation, the ergonomics experts could indicate problems related to the perceptual phenomena at work during the simulations. This allowed a better understanding and interpretation of the verbalisations and of the observed gestures of the FFs, and also made it possible go into greater depth on the results in an experiential approach to the processes of embodied and situated cognition at stake in these scenarios.

2.3.3. Methodological bias

32The objective of understanding the perceptual and cognitive factors involved in the integration of AR and how this affected experience led us to propose a specific methodology for accessing this experience data while ensuring that the study maintained a certain internal validity.

33Data collection aimed on the one hand to assess the performance of the evolving prototype within the framework of an activity finalised for design purposes, and on the other hand to study the transformations and adaptation strategies of the users.

  • 5 Test situations are described in part 4.2.1.

34Assessment of the device’s performance took place in two stages: firstly, we assessed the “passive” performance of the device, i.e. its capacity to enable participants to visualise and interpret the environmental elements useful for the reconnaissance of a complex space (the properties of which are very similar to operating environments). Secondly, thanks to the evolution of the prototype, we were able to assess the device’s “active” performance, in a more realistic5 simulation scenario, which allowed us to study its capacity to enable participants not only to understand a complex space but also to be able to progress within it and hence remain operational. For each of these steps, participants were invited to carry out simple tasks (search for hot spots) while simultaneously verbalising out loud (think-aloud protocol) what they were experiencing at every instant. Questionnaires were used after each scenario to collect the participants’ assessment of the device’s overall performance during the activity and its usability in terms of visual comfort, cognitive cost and learning cost (see Fig. 4).

35In order to carry out a more specific study of the way in which the integration of AR affected user experience, we set up an ad hoc methodology based on the principles of embodied cognition, with a view to collecting phenomenological data on the perceptions experienced. Here, the methodological difficulty lies in the object of study itself: the perceptual experience. On the one hand, it is often complicated for the participants to accurately describe their perceptual feelings, and on the other hand, because these phenomena are provided immediately after the experienced phenomenon, it is difficult to describe them afterwards, even with videos as a support, because they are often subconscious and volatile.

Fig. 4. Diagram of the methodological approach and data collected. 
Fig. 4. Schéma de l’approche méthodologique et des données collectées

Fig. 4. Diagram of the methodological approach and data collected.  Fig. 4. Schéma de l’approche méthodologique et des données collectées

36So in addition to the analysis of the think-aloud verbalisations and that of the semantic fields employed by the users in the questionnaires, the ergonomists’ participation in the tests also made it possible to rely on narratives of experience or, more precisely, on what Vermersch called the ’phenomenological explanations’ (Vermersch, 2006) that are necessary for a better understanding of the experience of the other, by going through a phase of ’experiential filling-in’ (ibid, p. 131). Furthermore, the study of gestures and pathways gave us an understanding of the body, not in terms of behaviour but in terms of choreography, so as to bring out phenomenological interpretations of the actions observed. This “choreographic” approach understands the body as being the result of gestures, which are themselves both expressive and instrumental (Baudin, 2021, in process). In this sense, the idea is not that the gesture is an extension of a thought or verbal expression, but rather an expressive phenomenon in itself, carrying with it social-cultural, biological and individual determinants, and endowed with a certain poiesis. Considered in this way, the gesture is a means to an end, in constant readjustment with the elements of the environment in which it takes place, according to a phrasing peculiar to each individual, which brings the perceptual and perceived dimensions into play.

37The study of situational gestures therefore meant observing not only the sequences of operations, but also the logics of movement, their distribution in space and time, the convocation of different body movements and bodily states (an expression derived from dance), i.e. the expressiveness that emerges from the gestures, depending on their tonus (see Fig. 5) or their poetics.

Fig. 5. Tools for collecting and analysing gestural and choreographic data. 
Fig. 5. Outils de recueil et analyse des données gestuelles et chorégraphiques

Fig. 5. Tools for collecting and analysing gestural and choreographic data.  Fig. 5. Outils de recueil et analyse des données gestuelles et chorégraphiques

(left: path study, right: study of times, gestural and postural notations). 
(à gauche : étude des parcours, à droite : étude des temps, notations gestuelles et posturales)

3. Thermal imaging in firefighting activity during interventions

3.1. Situations of firefighting intervention

38Initial studies took us to various fire stations and training centres. Although safety restrictions prevented us from taking part in real interventions, the interviews conducted with the various actors (professional firefighters, volunteers, instructors) and the observations made in fire stations and training centres (training sessions), allowed us first of all to gain a better understanding of the operation and organisation of fire stations (above and beyond differences between countries, which are organised slightly differently, we noted that organisation varies according to institutional/territorial attachment and hence to the role attributed to the fire station), to get an idea of how the FFs perceive their profession (courage, the motivation to save live, and the notion of vocation are points that often came up); and secondly, to identify the reference situations on which we would base our study.

39Furthermore, training forms the basis for the actual intervention. The strategies developed during interventions are very much governed by protocols that are established during training and then regularly practised in simulation. This is why the training and practice activities were a source of important data (see Fig. 6, showing field data). These protocols are essentially based on the definition of clear roles during interventions, each role being assigned tasks and intervention equipment. Many fire stations seek to ensure that every professional FF has polyvalent intervention-base skills, as the role in the field sometimes depends on where the FF is positioned in the fire engine and on the order of arrival at the scene.

40In order to identify the reference situation, several axes were taken into account. As part of our design approach, with the FFs we had to identify the different interventions in which thermal imaging was an important resource. Out of 9 European fire stations (Switzerland, France, Germany, Holland and England), 7 mentioned the utility of thermal imaging when searching for hot spots and for people. Nowadays this information is most often provided by a (portable) thermal camera. It is used almost systematically in the event of a fire intervention (residence, tunnel, basement, forest, hydrocarbons and chemical products) and when searching for people (car accident, drowning/shipwreck) or in the event of a gas leak.

Fig. 6. Observations and interviews carried out in training centres. 
Fig. 6. Observations et entretiens menés dans les centres de formation

Fig. 6. Observations and interviews carried out in training centres.  Fig. 6. Observations et entretiens menés dans les centres de formation

(Switzerland, on the left; England, on the right)
(Suisse, à gauche ; Angleterre, à droite)

41Moreover, from an analytical standpoint, and as the interviews progressed, it became clear that the FFs identified emergency fire interventions as being the most critical and where on-board AR thermal imaging technology would make the most sense, if only because these are situations in which the use of thermal imaging has become vital: “it’s our lifeline when entering and exiting a building”. The majority of FFs also recognised that in this type of situation, an on-board system would allow access to important visual information, while at the same time freeing up hands and thus, in their opinion, facilitating the intervention.

42Although it was not possible to carry out our observation under real conditions, the interviews and the follow-up of the training sessions were an opportunity, through observation of the protocols taught, to understand the role that imaging plays in intervention situations, and more precisely, to collect data on the different individual and collective intervention strategies, whether through mimed narratives or interviews conducted in this simulation context, which is conducive to the emergence of data for explanatory interviews. These field studies allowed us to obtain (subjective) experience data cross-referenced with the prescribed instructions and gestures and with the training narratives (Fig. 7).

Fig. 7. On the left, mimed gestures of progression within a space; on the right, observation of a fire simulation training session, learning to react to and withstand thermal phenomena. 
Fig. 7. À gauche, mime des gestes de progression dans un espace ; à droite, observation d’un entraînement de simulation incendie, apprendre à réagir et supporter les phénomènes thermiques

Fig. 7. On the left, mimed gestures of progression within a space; on the right, observation of a fire simulation training session, learning to react to and withstand thermal phenomena.  Fig. 7. À gauche, mime des gestes de progression dans un espace ; à droite, observation d’un entraînement de simulation incendie, apprendre à réagir et supporter les phénomènes thermiques
  • 6 To prevent incoming oxygen
  • 7 Instructor, La Rama training centre, November 2016.

43Whatever the context, fire interventions take place in unstable environments: variable light conditions (darkness, flames), dangers linked to thermal phenomena (flash-over, backdraft smoke explosion), disordered infrastructures (accident, fire, shipwreck), any factor that can change rapidly as the situation evolves. Smoke and soot impair visibility in unfamiliar places, which can then become difficult to interpret. Faced with an emergency, FFs must nevertheless act and react fast: to locate air inlets (windows, doors)6; to understand the scenography of the site (important for making an exit); to identify obstacles (their size, distance, nature), etc. All of this is necessary to complete the intervention and save lives, while at the same time protecting their own and those of their teammates. Even a few seconds can be crucial. It is therefore important to gain a rapid understanding of the location and to obtain stabilised information in order to make quick decisions. In such a context, stabilisation means understanding the dynamics of the fire, the fixed elements, the environment’s infrastructures, etc. Such stabilisation activity has become difficult: “in the old days most of the furniture was made of wood and wood does not make smoke; nowadays everything is oil-based, we have to cope with smoke, and smoke is dangerous, you can’t see anything”7. One of the strategies to be able to see in this atmosphere is “by lighting the smoke, you can get your bearings a little more, because then you can see the movement of the smoke, which always goes out through the exits”.

44Based on the observations and interviews we carried out, we were able to identify a clear intervention protocol, consisting of three distinct phases. First, a “reconnaissance” phase, the purpose of which is to search for hot spots (from the outside to begin with) and to gain an understanding of the environment, by probing it with the thermal camera. Then an exploratory phase where the firefighter progresses through the premises. These two stages are repeated successively (search followed by progression) until the firefighter reaches the source of the problem (fire source, gas leaks or other). He can then move on to the third phase, the actual “intervention” (extinction/repair), and later on to rescuing people. Firefighters’ expectations regarding thermal imaging in this type of intervention are summarised in the box below.

Thermal imaging and important elements of reconnaissance during firefighting interventions

Thermal imaging must allow FFs to recognise a building’s “hot spots”, structural elements and obstacles and make it possible to predict the dynamics of the fire and assess thermal phenomena. This reconnaissance phase is vital for carrying out an effective intervention: progress, put out the fire, spot gas leaks and save people’s lives.

So during an intervention it is important:
To be able to recognise:
 -  the main or residual source of a fire (thermal contrast)
 -  a gas leak (thermal contrast)
 -  a hydrocarbon leak (thermal contrast), in water for example
 -  a person’s body (thermal contrast and shape of contours)
 -  entry/exit paths (shape of contours)
 -  holes in the floor (shape of contours)
 -  obstacles on floor/ceiling (shape of contours)
 -  furniture (shape of contours)
 -  other elements of the environment (car, tree, lake, etc.)(shape of contours)
 -  where the water from the hose is landing (thermal contrast and natural vision)

In order to assess:
 -  the temperature and composition of the smoke (thermal contrast + natural vision)
 -  the temperature and composition of the flames (thermal contrast and natural vision)

45The use of thermal technology is generally taught to all firefighting personnel, but such devices remain expensive piece of equipment, mostly reserved for the incident commander and/or a response teammate. The thermal data provided by the camera is therefore shared both verbally and visually (by sharing the camera screen, see Fig. 8), thus avoiding air consumption. This equipment thus plays a full role in the circulation of knowledge and the sharing of information in order to ensure safety through teamwork. In all cases, thermal information is consulted on a one-off basis during the reconnaissance, progression and extinction phases.

Fig. 8. Postures for viewing thermal information during the reconnaissance phase. 
Fig. 8. Postures de visualisation de l’information thermique lors de la phase de reconnaissance

Fig. 8. Postures for viewing thermal information during the reconnaissance phase.  Fig. 8. Postures de visualisation de l’information thermique lors de la phase de reconnaissance

(on the left: gesture simulated during an interview, on the right: thermal visualisation). 
(à gauche : geste simulé lors d’un entretien, à droite : visualisation thermique)

3.2. Firefighter intervention strategies

46The protocols and strategies employed by firefighting teams are essential for limiting risks incurred during interventions and for ensuring individual and collective safety. The orders given by the incident commander allow him to guide the smooth running of operations and ensure that tasks are evenly divided (firefighting, equipment installation and perimeter security teams, etc.). But there are other aspects to interventions. Gestures, for example, which are ordered, sequenced and controlled. In order to identify hot spots and avoid obstacles as they progress, a firefighter proceeds systematically, using successive horizontal and vertical sweeps of the thermal camera as soon as he enters a new space (“volume”). He avoids obstacles by sweeping with his feet with arm outstretched for protection.

47By cross-referencing our field analysis with articles from the specialist press (Boyd, 2007), we identified three other elements that are crucial to ensuring the safety of firefighters during interventions: their ability to find their bearings in relation to the other members of the team, their ability to manage air consumption (“during interventions, you can consume up to 90l/min.” and their ability to find their bearings within the space.

48During the search phase, disorientation is a known issue for firefighters, especially in low-visibility environments, and this has led to specific search techniques. Always operating as a group, firefighters generally proceed by moving slowly along a wall, guided by voice or by holding on to a teammate’s jacket. These techniques are essential for finding one’s bearings within the space and in relation to other team members. They allow the firefighter to exit a building and, where necessary, provide teams outside the building with information about the location of the fire or people. During training exercises, volunteer firefighters in the Vaud canton (Switzerland) even place different colour lights throughout the space to mark the location of entry/exit points. Others use the right-hand wall strategy, guiding themselves along walls with their right hand and always turning right. Sometimes they may even use a white cane (as used by the visually impaired). Indeed, one FF told us: “we move around in the same way as a blind person; all senses are important, especially hearing, well that used to be the case, but now that we have ear protection, we can no longer hear”.

49While thermal imaging cameras help firefighters to visualise the environment in conditions of poor visibility, it is still difficult to assess the distance to obstacles. FFs have therefore developed strategies using their bodies, usually their hands and feet, but also their equipment (such as the hose) to “gauge” distances and reassess how far they are from surrounding objects. However, these difficulties make it harder to predict distances (and therefore risks), and this can be especially dangerous, particularly in cases where there is a limited supply of oxygen.

  • 8 Backdraft is the burning of superheated gasses that accumulate in an enclosed space and explode whe (...)

50Other techniques can allow the firefighter to anticipate and protect himself from thermal phenomena: using his natural vision, often with a torch, to assess the flammability of smoke (interpretation of smoke colour, composition and density); using his hand as a “thermal reference” to assess the temperature of an enclosed space (before opening a door, for example8). In all cases, he forces himself to rarely speak, preferring communication through gesture or graphics (thermal camera image) so as to avoid air consumption. Always carried out collectively, these strategies act as a complement to thermal information and can help the firefighter to stay alive.

51Thermal vision is thus one of the sensory aspects of the strategies deployed by FFs during interventions. Descriptions of intervention show that other modalities are constantly called upon in order to “see in conditions of poor visibility”. Hands, which perceive heat differentials through gloves (one commander commented that he had asked glove manufacturers absolutely avoid making gloves entirely permeable to heat); hands that provide guidance by following walls, that feel their way through a space, sometimes using canes, feet that stabilise within the environment and allow one to identify obstacles on the ground, hearing when ears are sometimes uncovered or when sounds are strident, that allow firefighters to “hear the fire go out”. These modalities combine with a thermal vision, nowadays directed at arm’s length, in a physical engagement that allows FFs to protect themselves while carrying out their interventions. Carried on the body, thermal vision implies a transformation of individual and collective bodily engagement and of the perceptual modalities in action. Using data from tests carried out in simulations, we studied the visual and perceptual dimensions at stake when integrating an AR device.

4. From experimentation to perceptual dimensions

4.1. Modelling the activity to identify simulation scenarios

52Using exploratory analyses of the reference situations, we established a schematic modelling of the use of the thermal camera in fire intervention (Fig. 9) with the intention of developing realistic simulations of standard situations. Our objective was twofold: i) to identify the best technical configurations and the specifications of the most suitable device, ii) to understand what the notion of “seamlessness” meant during interventions.

53This modelling allowed us to determine the variables to be taken into account when developing test scenarios in typical intervention situations, from both a contextual standpoint (organisational, material and environmental variation) and an individual standpoint (inter- and intra-individual variables relating to camera use).

Fig. 9. Schematic modelling of building fire intervention activity with thermal camera. 
Fig. 9. Modélisation schématique de l’activité d’intervention en incendie de bâtiment avec caméra thermique

Fig. 9. Schematic modelling of building fire intervention activity with thermal camera.  Fig. 9. Modélisation schématique de l’activité d’intervention en incendie de bâtiment avec caméra thermique

(adapted from Leplat & Cuny, 1977)
(adapté de Leplat & Cuny, 1977)

54The individual variables were represented using the already defined 7-person panel. For the contextual variants and invariants, we focused on the environmental conditions that appear to be the most critical for FFs when using the thermal camera to interpret the elements visualised.

4.2. Data from SAC (characteristic action situations) simulations during firefighting interventions

4.2.1. Simulation scenarios and summary of results

55The 7-person panel carried out two series of tests (2 to 3 hours for each series), with a 3-month interval. The first series of tests allowed us to study the perceptual issues by testing the AR V1 prototype during a simulated reconnaissance phase. The second series of tests allowed us to try an improved version of AR (V2) in a simulation of the progression and handling phase.

First series assessing “passive performance”: testing the device during the reconnaissance phase

56The first series of tests began on a static test bench, during which participants were asked to identify the best configuration for the V1 thermal AR prototype (see 1.3.1) in a scenario where they had to scan the space to find a hot target. They then had to select and verbalise the technical configuration that they felt to be the most effective in helping them to grasp the unknown environment - a windowless boiler room dotted with electrical control panels and pipes at different heights (fig. 10) - and identify the target (a kettle placed opposite the test bench but quite far away and slightly hidden), in a lighting environment that varied in accordance with the most commonly encountered cases (semi-darkness or total darkness) and with moving visual stimuli (person moving within the scenery).

Fig. 10. Boiler room, controlled environment for static tests. 
Fig. 10. Chaufferie, environnement contrôlé pour les tests en statique

Fig. 10. Boiler room, controlled environment for static tests.  Fig. 10. Chaufferie, environnement contrôlé pour les tests en statique

On the left: test room in full light (visible image), at the centre and on the right: obstacles and individual in the field-of-view in semi-darkness (thermal image). 
A gauche : salle de test en pleine lumière (image visible), au centre et à droite : obstacles et individu dans le champ visuel en semi-obscurité (image thermique).

57Then, equipped with the chosen configuration, the participants were asked to move around the room (in both dark and semi-dark modes), in which the kettle was this time hidden behind structural elements. We use the term “passive performance” because although the environmental conditions recreated certain parameters that are encountered during interventions (darkness, semi-darkness; unknown environment with obstacles and various hot spots), it was not a case of testing it in a finalised activity performance, but of testing the thermal vision provided by the prototype during the reconnaissance of an environment.

Second series of “active performance”: testing the device during the progression and intervention phase

58The second series of tests was designed to study in greater detail the perceptual dimensions of the device’s performance during its “active” role, i.e. during reconnaissance and progression in an environment with a view to intervening (manipulating the target). This test session took place in a military training area, which allowed us to get closer to the properties of an environment found in a real-life situation: unknown location, obstacles on the ground and at different heights, stairways, variable light conditions and poor visibility (Fig. 11).

Fig. 11. Military training area and bunker, used for tests in realistic conditions. 
Fig. 11. Zone d’entrainement militaire et blockhaus, utilisés pour les tests en conditions réalistes

Fig. 11. Military training area and bunker, used for tests in realistic conditions.  Fig. 11. Zone d’entrainement militaire et blockhaus, utilisés pour les tests en conditions réalistes
  • 9 Bullard Eco X (2.75’ screen, 240x180 px resolution, HFOV: 60°, High: 60°, Low: 40°)

59Once again, the participant’s task was to find a hot spot and bring it back to the starting point of the test. This task allowed us to simulate both the search for the hot spot and the reconnaissance of the site, the progression using thermal information, and also the handling and orientation within the space (return to the exit). The participant thus had to perform this task with a portable thermal camera9 and with the AR prototype, in relation to two factors (site and visibility) and in three different contexts: once inside a bunker in the dark, once inside another bunker filled with white smoke (simulation of building fires), once outside during the day (simulation of reconnaissance from outside a building).

60We thus tested the active performance of the device, i.e. its capacity to meet expectations regarding thermal imaging in a simulation; while the latter was of course invented, its objective and conditions used the same components found in real intervention situations.

4.2.2. Results: the perceptual and cognitive dimensions at stake

The “reconnaissance” phase

61Table 1 below summarises the configuration choices and excerpts from the related verbalisations on which we based our approach to the perceptual processes at work during the activity, in order to a) understand how the FFs used the visual resources to reconnoitre premises and b) to identify the sensory mechanisms “disturbed” by the integration of the AR prototype.

  • 10 Intermediate, 26.06.2019, Neuchâtel (boiler room)
  • 11 Expert, 19.03.2019, Neuchâtel (boiler room)

62In the first test, carried out on the static test bench, the unanimous choice (7/7) of the Boson camera which provides a visual angle of 92° (the Lepton 56°), extends the natural perceptual capacity, the binocular field of vision of which reaches 62°. Although this choice generates a visuospatial disturbance (Fuchs, 2018, p304), it was made on the basis of its performance during the visual reconnaissance of sites. It provides an overall view of the scene “at a glance10, which prevents FFs from forgetting zones within the scene, especially corners of ceilings which can be dangerous (“they can create air intakes and rekindle a fire”). “It lets you see more and be more careful than with a standard camera where you might miss certain elements11.

63This choice shows that what is required is not an “enhanced reproduction” of what is naturally perceived, but instead an “autonomous” image that optimises information intake. With this choice, image quality (sharpness, precision, contrast) is essential, not only for recognition but also for simultaneously identifying and interpreting the different elements of the environment (see verbalisation table: “You can see the conduction of heat on the pipes, and that’s important. You can also see the reflections on certain surfaces”). So it is because it provides better image quality that the Boson camera was chosen by all participants.

64The visual information chosen during this phase suggests that primacy is given to the discernment and understanding of visual information in relation to one’s location, since the choices favour a proposition where the location of the object differs from the natural referent (wider field of vision). However, the presentation of this information in the perceptual field must be as close as possible to normal perception or at least allow an optimal conjugation with normal visual perception and above all not disturb it.

65Regarding image capture and projection, most of the panel (6/7) preferred a capture configuration, centred in relation to the eyes and projecting the same image onto each eye (“false binocular”). This configuration is the one that is closest to an egocentric referent of visual perception, allowing one to situate oneself in space, with distance and depth adjustment being achieved via the natural mechanism of egocentric referencing (see table, verbalisation: “with the camera above, I can see my two hands”).

Table 1. Summary of the results of the static tests of phase 1 - choice of configuration. 
Tableau 1. Synthèse des résultats des tests statiques de la phase 1 – choix de configuration

Table 1. Summary of the results of the static tests of phase 1 - choice of configuration.  Tableau 1. Synthèse des résultats des tests statiques de la phase 1 – choix de configuration
  • 12 Expert, 19.03.2019, Neuchâtel (boiler room)
  • 13 This point has been demonstrated in numerous works of research, but worthy of particular note is th (...)

66This image capture in the visual axis is reinforced by the choice of a binocular projection (the same image on both eyes creating a false binocular). This choice makes it possible to reproduce a sensation of depth, but without being in an entirely stereoscopic mode. Due to the perceived discomfort (even nausea), the participants unanimously rejected the monocular configuration which projects the image onto only one eye: “It’s unsettling”12. Indeed, the reception of images of different registers in each eye (brightness, patterns) causes perceptual problems such as binocular rivalry, alternating vision or permanent suppression13.

  • 14 Novice, 12.03.2019, Neuchâtel (boiler room)

67In the following tests, designed to test this choice in a more dynamic scenario of active research in space, it was confirmed that a vision close to natural stereoscopy (Boson positioned on each side of the head at eye level, with each image being projected onto each eye), chosen by just one participant, was a factor of discomfort for the majority, as it made it more complicated to interpret depth because it accentuated other perceptual problems: difficulty in merging images in the near field and increasing the distance of objects in space: “in 3D, the cameras do not sufficiently adjust to the focus of your eyes. Objects appear very small and distant compared to reality”14.

68The choice of a wide angle of vision was confirmed as being a relevant choice for the type of active view that is required during this phase: “What I think is good is being able to sweep and look for the points that interest me” (expert).

  • 15 Expert, 19.03.2019, Neuchâtel (boiler room)

69This active “sweeping”, not only allows FFs to move their eyes and head – which are used to help find information within in the environment - but also provides a sort of attentional filter when exploring premises – allowing FFs to “look for the points that interest me”15. It should be noted that this filter becomes increasingly important as expertise grows, since it is learned in training and then practised. The sweeping movement that the FF describes shows that the site reconnaissance phase requires a certain attentional flexibility when taking in visual information in order to stabilise the determining elements of the spatial situation and thus to be able to anticipate progression (“to see the most direct paths towards the fire source and/or the person”) and intervention (“you have to be able to locate the smoke evacuation zones”). Perceptual attention is then both focused and selective, with the objective of searching for hot spots and then for obstacles and structural elements in the environment, engaging in a permanent visual sweep so as to be able to quickly capture the environment as a whole and then select the points to be identified. In this respect, the use of the wide visual field that makes it possible to bring elements of a naturally peripheral vision into the focal field and make them visible was considered an advantage by all participants because it saves time, and in some way frees up the dynamic of progress in space: “We very rapidly sweep a very large area because it is in our field of vision. That’s a big plus. We really move more freely” (expert).

70However, the need to use one’s hands or arms to gauge the image tends to mean that while the device would appear to facilitate the search, the orientation and anticipation required during this site reconnaissance phase may not be entirely optimised.

  • 16 The Moverio screen used during S.A.V.E. and projecting a virtual image at a distance of between 2m (...)

71So depending on the type of lighting environment, the perception of the thermal image is not used in the same way (see Table 2). In the semi-dark scenario, where certain elements of the site could be seen naturally, all of the participants experienced a permanent visual “to-ing-and-fro-ing” between the thermal image view and the natural view of the environment, made possible by the characteristics of the projection screen16 in the glasses allowing them to capture the environment, either below or to the side of the projected image. This point validates research carried out in the psychology of perception, which shows that subjects cannot help directing their attention towards the place occupied by a peripheral cue (even if it is not informative), in order to complete the visuospatial attention process and hence the spatial interpretation of the cues captured (Delorme, & Fluckiger, 2003, p. 440). Conversely, in the total darkness scenario the visual perception of the environment was essentially fed by the thermal image; the feeling was thus very similar to a search in a virtual non-augmented environment. In such contexts the orientation and anticipation processes seemed a little different, and above all required compensation strategies that were more or less exacting, depending on the participant’s expertise. So although the time it took to perform tasks in semi-darkness was longer, whatever the level of the participant, the participants’ evaluation of visual and cognitive fatigue shows that the integration of AR during the reconnaissance phase had a certain sensory cost for the least experienced among them (see Table 2). The eye movements and focal adjustments needed to combine thermal image with environmental image required ocular gymnastics which might be disruptive (even nauseating) when one has not had enough training.

Table 2. Summary of the results of the site reconnaissance tests in phase 1. 
Tableau 2. Synthèse des résultats des tests de reconnaissance du lieu de la phase 1

Table 2. Summary of the results of the site reconnaissance tests in phase 1.  Tableau 2. Synthèse des résultats des tests de reconnaissance du lieu de la phase 1

The “progression” phase

72The second series of tests allowed us to examine the device in environmental conditions that were somewhat more complex (and thus closer to real-life situations) and assess its performance not only during the reconnaissance phase but also during a progression phase, where it is no longer just a matter of understanding the environment in order to stabilise its elements and be able to anticipate, but also one of being able to find one’s bearings by constantly readjusting one’s bodily engagement to suit the different environmental, thermal and operational dynamics. In this series of tests, the prototype had been improved, in particular to allow participants to choose different thermal image processing configurations to suit the light conditions encountered in the scenarios (see Table 3).

  • 17 Expert, 27.09.2019, Colombier (fire brigade training camp), outdoor condition

73The device demonstrated its limits in these dynamic situations with regard to discerning and understanding thermal information. Despite the possibility of changing the thermal image configurations to suit the thermal scale of the elements in the environment, in the dark the image contrast was considered insufficient to perceive certain obstacles on the ground. Outdoors, the image was not sufficiently visible due to the luminosity, which generated an additional cognitive load: “it requires active concentration to force the eye to work with the device17. Some participants moved into the shade or put their hands behind the screen to improve the visibility of the virtual image. The adaptive aspect of the available thermal modes caused a change in contrast in the image as soon as a hot element appeared in the scene. The fact that this adaptation occurred on the basis of thermal and not environmental data was considered dangerous in as much as the image contrasts were no longer sharp enough to discern obstacles on the ground. Moreover, this adaptation of contrasts disrupted the participants’ spatial reference points and required an additional effort to interpret the scene. “It is not easy to move forward. You can’t see much, there aren’t many thermal variations in the environment” (expert). As the thermal information provided relies on colour gradients alone, the participants lacked information that would allow them to determine the temperature of the hot spots (temperature scale or sight) and hence to interpret and anticipate the situation. This point demonstrates the need for an image that simultaneously allows firefighters not only to discern but also to understand what they are seeing. In all of the scenarios the lack of image contrast and sharpness led the most experienced to feel the need for additional information to interpret the image in an action dynamic: “it’s effective for identifying hot spots but not for understanding the environment” (expert).

  • 18 “Salience is first and foremost the surfacing of a form from a background, i.e. an element being pa (...)
  • 19 Novice, 18.03.2019, Neuchâtel (boiler room)

74In addition to these issues of visual salience18, during this dynamic phase the participants were also able to experience the latency effect of the thermal image in relation to the real data, linked to the sampling frequency of the camera (60Hz for the Boson chosen). When moving or during rapid head movements, the image could sometimes appear jerky “when I turn, I have to wait for the thermal [image]19. This point was mentioned by several participants and is important, because this temporal shift, which generates a visuo-temporal incoherency, would seem to be more of an issue than the spatial shift, which is compensated by using hands and arms as a reference point. This issue had already been highlighted in a study carried out on an AR application for learning (Svétoslavova, Burkhardt, Mégard, & Ehanno, 2007).

75In these dynamic tests the camera’s angle of vision, increasing the field of vision compared to natural vision, continues to be seen as an advantage as it allows the user to “see into the nooks and crannies” without having to look for visual information. This said, the limitations of the device mean that it provides a virtual image that is not merged with the real image, and this biases the depth effect and hence the evaluation of distances. This can be particularly unsettling in a dark or smoky environment where the user relies mainly on the virtual image. The participants compensated for this bias by using an egocentric (arm/hand) reference frame visible in the virtual field to gauge their distance from real objects:

The distance discrepancy can be grasped and learned much faster than I thought. Initially, I tried to superimpose, but then I used the wide angle. When I make movements with my hand, it’s not a problem” (novice).

76In this reference strategy, the wide visual angle is also important for helping with orientation “it’s important to see your hand, your foot in the thermal image to be able to gauge distances” (expert). We find our bearings mainly in relation to the ground, which allows us to situate ourselves in relation to the direction of our postural balance, a dynamic of interaction with our environment which enables us to perceive and control our orientation. However, although vision allows us to reinforce this capacity, it is only in an inter-sensory modality shared with vestibular and proprioceptive systems (Delorme, & Flückiger, 2003).

Table 3. Summary of the results of the dynamic tests of phase 2. 
Tableau 3. Synthèse des résultats des tests dynamiques de la phase 2

Table 3. Summary of the results of the dynamic tests of phase 2.  Tableau 3. Synthèse des résultats des tests dynamiques de la phase 2
  • 20 Novice, 18.03.2019, Neuchâtel (boiler room)

77Another strategy that was the object of much comment during these dynamic tests was the use of “ocular exit”, which is provided by the quality of the screen and the position and size of the projected image, freeing up a natural field of vision. This point demonstrates the need, via natural vision, to acquire the additional information that is required during the progression phase, to allow direct capture of clues conducive to orientation and to the localisation of objects, as well as bodily adjustment. “If you look to the left or right, ‘it disappears’, you can concentrate on something else. It’s quite good when moving forward” (expert). “I thought I would have more difficulty switching between the virtual and the real20. Yet during this dynamic phase, the participants were more aware of the effects of a certain visual fatigue. The information projected onto a fixed two-dimensional focal plane can cause vergence-accommodation problems and induce visual fatigue: “it feels like a cinema screen

78Finally, note that in these dynamic test scenarios in smoky and dark conditions, some participants relied solely on thermal vision. In this case, the on-board device’s hand-release option is an advantage clearly highlighted by the experts. Feet, hands and arms thus play an active but not always sufficient role in understanding and orienting in space: two participants, one expert and one novice, became disoriented.

79The dynamic tests thus made it possible to confirm that the sensory resources in action are not only visual, but draw on global sensory-motor perception (or synaesthesia) where the representation of space and movement for orientation uses different perceptual (gravitational, visual, egocentric) (Delorme, & Flückiger, 2003) and sensorimotor reference points in an articulation of different sensory modalities (the importance of hearing and touch highlighted by the FFs during the field studies, and transcribed in the brief description of intervention situations in part 3. 1) that the integration of AR must accompany without hindering.

5. Bodily engagement

5.1. New forms of physicality

  • 21 The order was randomly chosen for each participant in order for the chronology for using each devic (...)

80During the dynamic tests, each scenario was carried out by the participants using a thermal camera and the S.A.V.E. prototype21, thus making it possible to compare the bodily engagements generated by each device, in particular by collecting sequences of gestures and verbalisations, as well as the gestural flows observed in the thermal recordings we made (see Fig. 12).

81When using the Bullard thermal camera, on which they have been trained, the experts spontaneously set up the sequenced and methodical dynamic they had learnt and been taught. As soon as they arrived in a new space (“volume”), they placed themselves in a stable position, back against a wall or just at the entrance, to carry out reconnaissance using successive lateral and vertical sweeps involving the arm and torso (see Fig. 13). These gestures, acquired through practice and training, were precise, orderly and unitary, unlike those of the novices who performed a multitude of successive and/or simultaneous movements. The pauses taken by the experts to scan the environment were relatively long (~25s) so as to properly stabilise the elements of the scene. Progression through the space then took place by walking along a wall using the other hand, and sweeping with the foot to avoid obstacles.

Fig. 12. Tools for analysing verbalizations, gestures and flow of gestures. 
Fig. 12. outils d’analyse des verbalisations, gestes et flux de gestes

Fig. 12. Tools for analysing verbalizations, gestures and flow of gestures.  Fig. 12. outils d’analyse des verbalisations, gestes et flux de gestes

82On the other hand, novice and intermediate participants had a less assertive and more opportunistic approach. Reconnaissance and progression were carried out in a more blurred and less sequenced manner. Pauses to sweep the environment were shorter and sometimes occurred right in the middle of a space. Gestures were less methodical. Progression was achieved by looking at the screen. Some FFs stopped using the thermal camera and relied more on their natural vision to move around. The steps were shorter, less stable, slower and less precise than those of the experts.

Fig. 13. Search strategies with thermal camera. 
Fig. 13. Stratégies de recherche adoptées avec caméra thermique

Fig. 13. Search strategies with thermal camera.  Fig. 13. Stratégies de recherche adoptées avec caméra thermique

(on the left the experts; on the right the intermediates and novices) 
(à gauche les experts ; à droite les intermédiaires et novices)

83When using the S.A.V.E. device, experts and novices used a similar strategy: reconnaissance and progression were more simultaneous (Fig. 14). There were fewer pauses to sweep the surrounding space, replaced by micro-stops (~2s) used continuously throughout the progression.

Fig. 14. Search strategies adopted by both experts and novices with S.A.V.E. 
Fig. 14. Stratégies de recherche adoptées à la fois par les experts et les novices avec S.A.V.E.

Fig. 14. Search strategies adopted by both experts and novices with S.A.V.E.  Fig. 14. Stratégies de recherche adoptées à la fois par les experts et les novices avec S.A.V.E.
  • 22 Expert, 19.03.2019, Neuchâtel (boiler room)

84The system, which was tested twice by the participants, would therefore appear to blur the expertise of professionals and turn novices into “thermal imaging experts”. This quality does not however make them experts in the activity. The professional experience of the experts still makes them perform far better when carrying out their tasks (see task execution times, tables 2 and 3). Yet as far as general choreography is concerned, there is little difference in expertise levels and progression appears to become more “risky”. The use of the S.A.V.E. system, which gives the user more freedom of movement and “visual autonomy”, seems to change the way the professionals see and find their bearings. “I no longer do systematic scans when searching, because the field of vision is wider. I have more confidence, but if there had been more obstacles on the ground it would have been better to do it22 said an expert and instructor.

  • 23 Expert, 27.09.2019, Colombier (fire brigade training camp), white smoke conditions
  • 24 Expert, 30.09.2019, Colombier (fire brigade training camp)

85Both the professionals and the novices opted for a search strategy in which exploration and progression in space occurred simultaneously and in a more “natural” and less cautious manner: they did not necessarily walk along a wall, they performed more head rotations and 360° turns to scan the environment in a dynamic that led them directly to the heart of the scene (see Fig. 15). These rotations, at the expense of systematic scanning from top to bottom and from right to left, tended to cause them to lose their bearings within the space. As one expert in white smoke conditions told us: “I could no longer find my bearings in relation to the hot spot23. Furthermore, most of the participants admitted having “trusted the thermal image24 in situations of total darkness and thick smoke. In these contexts hands were not used to gauge distances, but rather to recognise by touch the elements discerned from the thermal image. In this case, hands helped not with orientation but with an understanding of the visualisation, which led two participants to find themselves in a situation of disorientation in white smoke conditions.

Fig. 15. Notation diagram for the “choreographies” adopted by users, comparison of the use of the thermal camera (CT) and the AR device depending on scenario and expertise. 
Fig. 15. Schéma de notation des « chorégraphies » adoptées par les utilisateurs, comparaison de l’usage de la caméra thermique (CT) et du dispositif de RA selon les scénarios et l’expertise.

Fig. 15. Notation diagram for the “choreographies” adopted by users, comparison of the use of the thermal camera (CT) and the AR device depending on scenario and expertise.  Fig. 15. Schéma de notation des « chorégraphies » adoptées par les utilisateurs, comparaison de l’usage de la caméra thermique (CT) et du dispositif de RA selon les scénarios et l’expertise.

5.2. Body protection

86During the field studies we noticed that over the course of the extinction phase, the firefighters of the “lead duo” are usually on their knees to protect themselves from thermal phenomena. This learned and practised gesture is particularly important for protecting oneself from backdraft (explosion of superheated gasses when there is an intake of air, most often located at height). In this kneeling posture, the firefighter carrying the thermal camera holds it with his or her body bent over and stable, arm outstretched, in order to show the image to his or her teammate, thus creating a “zone of protection” from the surrounding danger with the extension of the arm (see Fig. 16).

87Due to its on-board nature, the S.A.V.E. device involves carrying visual information on one’s body (proximal zone) rather than at arm’s length (distal zone). Firefighters must therefore look for information with their heads, thus exposing the vital functions of the body, where the body tends to create a protective dynamic by extending the arm. The thermal camera makes it possible not only to anticipate a thermal risk but also to communicate this anticipation by showing it to team members if so required. The on-board dimension eliminates these two functionalities. One’s hands become free, of course, but the perceptual attention paid to the thermal image in the visual field tends to focus attention on the interpretation of this field, while at the same time generating a progression dynamic during which tactile information essentially becomes a mode of localisation rather than one of protection (see Fig. 16). All of the participants tended to explore with their arms and hands while at the same time using visual discovery to assess distances or understand objects without anticipating (or rarely anticipating) the risks associated with contact with objects and without seeking to orient themselves safely, but instead to stabilise the visual information received.

Fig. 16. Body protection with deported image (left) and worn on self (right) (same participant). 
Fig. 16. Protection du corps avec image déportée (gauche) et portée sur soi (droite) (même participant)

Fig. 16. Body protection with deported image (left) and worn on self (right) (same participant).  Fig. 16. Protection du corps avec image déportée (gauche) et portée sur soi (droite) (même participant)
  • 25 This point is mainly based on post-test verbalisation data, but also in part on experience narrativ (...)

88In spite of the possibility of an ocular exit with the SAVE device, the strength of the image embedded in the visual field tends to guide the perceptual modalities at play25, with the purpose of the search for sensory information being to complete image interpretation rather than to understand the environment and improve firefighters’ ability to find their bearings. In this respect, an “attentional tunnelling” effect (Pinet, 2011) occurs, i.e. a fixation on a particular element and forgetting one’s environment. It is clearly difficult to apply this last point to all AR technology, as it is the result of analyses carried out on an unsuccessful prototype. These results nevertheless tend to confirm those of research carried out in different fields of AR application. In particular, in a study on the application of AR to the urban mobility of senior citizens, the authors show that the use of AR glasses to assist pedestrian navigation causes participants to direct their constant and almost exclusive attention to virtual information (Montuwy, Cahour, & Dommes, 2019).

  • 26 During fire interventions FFs are equipped with masks and with bottles containing between 6 and 9 l (...)

89There are two principal dangers for firefighters when perceptual resources are monopolised in this way: disorientation and lack of oxygen. Knowing how to manage air consumption is essential to firefighter survival. When there is no visibility, it is by gauging the distance to the fire that firefighters are able to assess whether or not they have enough air26 to engage (intervention in a tunnel for example). If the on-board thermal image does not itself allow firefighters to assess distances, and moreover requires tactile engagement to do so, the danger then remains present and is even increased, because the on-board (and therefore individualised) image does not permit a collective evaluation of the danger and leads to additional body movements which are costly in terms of oxygen consumption.

5.3. New registers of intelligibility

90Reconnaissance and progression activities require the construction of a spatial representation. With an embodied approach, this representation is not simply a cognitive and amodal construction, but, as we can see from the empirical results, it is constructed through the physical exploration of this space, requiring an articulation of the sensory and sensory-motor dimensions.

  • 27 Expert, 30.09.2019, Colombier (fire brigade training camp)
  • 28 In this respect our results are similar to observations made with regard to the integration of AR i (...)

91In this respect, the “confidence” in the thermal image mentioned by the participants is interesting. It reveals a certain form of perceptual dependence on the information collected. Yet during the tests carried out in darkness and smoke, where recourse to natural vision was hardly possible, most of the expert participants mentioned their difficulty in interpreting the thermal data and the risks that this could cause in a real situation, while admitting that they had essentially allowed themselves to be guided by the technique. “The thermal image helps to identify heat-related danger, but not the danger posed by obstacles. I had to use my feet and hands to help me”, “it’s unsettling, because you think you can see the obstacles27. So while the device does indeed let FFs “see the invisible”, it does not always allow them to understand what they are seeing, and can even obstruct clues provided by natural vision, that would let them understand the environment. This point is illustrated in particular by the necessity of the ocular exit. Furthermore, interpretation of the thermal data on surrounding objects requires cognitive adaptation; while natural perception is based on proprioceptive visual data such as volumes, contours, textures, shadows or reflections that are not easily discernible by current thermal imaging, a thermal image must be interpreted by symbolic data that essentially play on colour gradation28. With the S.A.V.E. device, forms of comprehension of space therefore work on perceptual-cognitive registers other than those usually implemented, giving rise to new forms of intelligibility of space, based on theoretical (the temperature differentials of certain materials) rather than experiential inferences. However, in order to optimise this exploration of space and to favour immediate perception-action, hands and feet act to compensate, submitting themselves to the search for clues that might complete the missing information with the image. In this respect, bodily engagement during these phases of experimental reconnaissance and progression is essentially directed towards the interpretation of visual stimuli.

  • 29 “Spatial representation” refers here to the way in which an individual understands the elements of (...)
  • 30 “Affordance specifies the actions that may potentially be applied to the elements of an environment (...)

92By changing the bodily forms of space exploration in this way, the device modifies the construction of spatial representation29 (Dutriaux, & Gyselinc, 2016) at the heart of reconnaissance and progression activities. It changes the perception of space by disrupting the relationship of affordance30 (Gibson, 1979) which allows us to understand the environment in a relationship of action. During the first test, for example, all of the participants opted for the device which allowed them to obtain the sharpest, most precise and contrasted image, in a dynamic of “vision to act” - “to be able to sweep for the important points” - and not of “vision only to perceive”. Yet depending on the environmental conditions, the image, while often unavoidable, lacked either visual or temporal quality (see description of intervention situations in part 3.1), forcing the participants to compensate by proprioception, touch and posture. If sight does not provide clues for action, it is the body which will seek it in a more proximal modality (touch and kinaesthesia being proximal senses), impeding the anticipation necessary for the intervention and leaving aside the somatosensory engagement of the body in its more distal capture of space. Guiding actions by sight encourages us more to validate or assess what has been seen rather than to explore the environment as we perceive it.

93By placing a thermal image in the direct perceptual field, we disrupt what Merleau Ponty called the “haptic gaze” (1945), the gaze which not only allows us to understand the environment in the capacity for action that it creates, but also the gaze which enables us to situate ourselves in the world, in the environment. With the integration of AR, a new form of visualisation is engaged, forcing us to find new compromises between sensorial perception and coherent representations (Boukho, 2011), a toing-and-froing where the body will use plasticity to interpret the environment with its naturally haptic vision and the thermal image completed with the tactile and proprioceptive images of hand-and-foot engagement. However, as we saw during the field studies, the body-in-intervention is often a collective body, a body that acts by means of and through the group.

5.4. And the collective body

  • 31 Firefighter’s verbalisation.

94Although we were unable to address the group’s work within the framework of this project, mainly due to the experimental system being in its infancy, our observations in reference situations, the interviews we conducted and our analysis of professional articles allowed us to make initial observations on the challenges firefighters face when using AR. In particular, AR tests carried out in 2007 in live situations (Boyd, 2007) show that the organisation and the hierarchy established during interventions are profoundly transformed by the implementation of AR. In this study, each firefighter was equipped with a thermal AR device. It was seen that each firefighter “read” the fire in a different way: one dealt with extinguishing the fire while the others analysed its direction or looked for residual fires. Unlike the way a team leader usually functions, responsibility was shared. The improvement to each individual’s ability to see reduced the officer’s ability to manage and coordinate his team. The firefighters moved more rapidly within the structure, without necessarily following the progression techniques (using walls to maintain their orientation) that normally allow an officer to give directions to firefighters to guide them out when they are in difficulty. These new dynamics affected the officer’s ability to maintain control and responsibility for his team. The collective spirit is fundamental to the profession: “staying together means staying alive31, especially during firefighting interventions (holding a team-mate’s jacket, working in single file, vocal contact). The use of AR changes the way the group operates and tends to define more individual modes of action.

6. Conclusion

95“Making the invisible visible” is one of the objectives of AR in many fields. However, the first results of the project presented here show that the integration of this technology is not limited to a modification of visual perception intruding into a perception-decision-action loop, but calls for a deeper mutation of in situ bodily engagement to achieve an immediate understanding of the environment. In turn, this transformation of bodily engagement also implies a different construction of the representation of space.

96Our methodological approach, combining “traditional” analysis data with an analysis of gestures allowed us to tend towards an embodied cognition approach. This focus on body gestures let us show the adaptations implemented by the participants in order to make the best possible use of the AR device in relation to specific places and luminosities. In these adaptations, we noted in particular the strategies developed to gain an understanding of the environment in a relationship of affordance that can lead to risky behaviour.

97These results are similar to the conclusions of other studies which show how, by incorporating new visual data into the perceptual field, not only are visual ecological dynamics (Charbonneau, 2010) and thus expert interpretations (Arnaldi, Guitton, & Moreau, 2018) put to the test, but more generally so are our forms of arrangement between different sensory modalities. By introducing new virtual information from the environment, we are not perceiving a second environment, but a mixed environment that combines the virtual and the real that, in a given situation and through our senses, we are trying to grasp (Casiez, Granier, Hachet, Lepetit, Moreau, et al., 2018, p. 245‑246) and make coherent. However, when there is a discrepancy between the eye and the body, our modes of sensory-motor coordination can be disrupted (Boukho, 2011). In the artistic corpus, several publications have led to observations on the perceptual upheavals caused by new immersive environments. In particular, Boukho’s analysis of artistic installations testing the link between the visual universe and walking, refines our understanding of what is at stake in this alteration of the correlation between visual and kinaesthetic perceptions of walking: when the seen movement differs from - or is out of sync with - the movement perceived through the steps taken, the vestibular system is disturbed, affecting one’s sense of direction, and one’s haptic and proprioceptive senses. In such cases we refer to a “pseudo-haptic” phenomenon in which visual perception becomes the most important modality for negotiating this new mixed environment. What some call visuo-haptic interference (Fuchs, 2018) is experienced as new perceived environments, which the body in its perceptual and motor-based interpretation will always try to understand in terms of the action. During the tests, the various verbalisations of the expert FFs would seem to indicate that trained bodies can compensate and calibrate themselves in relation to these new stimuli, if the image retains an egocentric dimension. The adaptation therefore takes place in a certain plasticity of the body (extension of sight through hands, feet, movements) so as to understand the space in its haptic dimension. However, it is also to the extent to which the technical devices also manage to come close to a haptic vision (a vision that facilitates the relationship of affordance) that the adaptation will be total. In this sense, it is in a relationship of co-plasticity between the artefact and the body that a new mixed reality will certainly become fully operational.

Haut de page

Bibliographie

Albarelli, A., Celentano, A., Cosmo, L., & Marchi, R. (2015). On the Interplay between Data Overlay and Real-World Context Using See-through Displays. Proceedings of the 11th Biannual Conference on Italian SIGCHI Chapter. https://doi.org/10.1145/2808435.2808455

Arnaldi, B., Guitton, P., & Moreau, G. (2018). Réalité virtuelle et réalité augmentée : Mythes et réalités. Londres: ISTE éditions (Collection Informatique).

Baudin, C. (2021 - ongoing). Mise en abîme du « corps inspiré » : L’urgence du retour au sensible. Actes du 55e Congrès SELF « L’activité et ses frontières : Penser et agir sur les transformations de nos sociétés ».

Baudin, C., & Nusshold, P. (2018). Digitalization and the Evolution of Real Work: Introduction. Laboreal, 14 (2). https://journals.openedition.org/laboreal/pdf/583

Baudin, C. (2017a). Au cœur du sensible : L’or « profane » de Madre de Dios. In Philippe Geslin (dir.). L’anthropotechnologie : Cultures et Conception pp. 71‑101. Londres: ISTE Editions.

Baudin, C. (2017b). La perte des sens et du sens des corps technicisés. Actes FISO (Festival International de Sociologie), La fabrication des corps au 21e siècle-. p. 1‑15. https://www.academia.edu/35272958/La_perte_des_sens_et_du_sens_des_corps_technicisés

Baudin, C. (2016). Concevoir du sens par les sens. Actes du Sensolier « Design sensoriel. Comment tenir compte de l’expérience multisensorielle de l’utilisateur/consommateur ? ». p. 1‑6. https://www.academia.edu/34216156/Concevoir_du_sens_par_les_sens

Baumeister, J., Ssin S.Y., ElSayed N.A.M., Dorrian, J., Webb, D.P., Walsh, J.A., & al. (2017). Cognitive Cost of Using Augmented Reality Displays. IEEE Transactions on Visualization and Computer Graphics, 23 (11), 2378‑2388.

Béguin, P. (2004). L’ergonome, acteur de la conception. In Pierre Falzon (dir.). Ergonomie. pp. 375‑390. Paris: Presses Universitaires de France.

Bernard, M. (2001). De la Création chorégraphique (1ʳᵉ éd.). Paris: Centre National de la Danse.

Berthoz, A. (1997). Le sens du mouvement. Paris: Editions Odile Jacob.

Bobillier-Chaumon, M.-E., (2017). Du rôle des TIC dans la transformation digitale de l’activité et de la santé au travail. La Revue des Conditions de Travail, ANACT, 6, 16‑24.

Boukho, C. (2011). Les parcours immersifs et l’expérience quotidienne de la marche. Revue Réel/ virtuel : enjeux du numérique.2(2). https://biblio.ugent.be/publication/8124031/file/8134276.pdf

Boyd, J. (2007). Using Hands-Free Thermal Imaging Cameras. Fire Engineering Magazine, 160 (5), 95‑97.

Casiez, G., Granier, X., Hachet, M., Lepetit, V., Moreau, G., & Nannipieri, O. (2018). Vers des EV plus liés au monde réel. In B. Arnaldi, P. Guitton et G. Moreau (dir.). Réalité virtuelle et réalité augmentée : Mythes et réalités. pp. 231‑259. London: ISTE Editions.

Charbonneau, M. (2010). Approche méthodologique et comparative des critères de qualité d’image, de perception et d’exploitabilité opérationnelle : application aux systèmes d’aide à la vision nocturne en aéronautique. Thèse de doctorat, Sciences cognitives, Université de Bordeaux 2.

Claverie, B., & Léger, A. (2009). Vision augmentée par HUD : pour une prise en compte des contraintes psychophysiologiques. Revue des Interactions Humaines Médiatisées, 10 (1), 27‑58.

Daniellou, D., & Béguin, P. (2004). 20. Méthodologie de l’action ergonomique : approches du travail réel. In Pierre Falzon (dir.). Ergonomie. pp. 333‑358. Paris: Presses Universitaires de France.

Delorme, A., & Flückiger, M. (2003). Perception et réalité : une introduction à la psychologie des perceptions. Paris: Ed. De Boeck (Collection Neurosciences et Cognition).

Dutriaux, L., & Gyselinck, V. (2016). Cognition incarnée : un point de vue sur les représentations spatiales. In L’Année psychologique.116 (3), 419‑465.

Fuchs, P. (2018). Les enjeux et les risques de la démocratisation de la RV-RA. In Bruno Arnaldi, Pascal Guitton et Guillaume Moreau (dir.). Réalité virtuelle et réalité augmentée : Mythes et réalités. pp. 299‑311. London: ISTE Editions.

Garçon, A.-F. (2012). L’imaginaire et la pensée technique. Une approche historique, XVIe-XXe siècle. Paris: Ed. Classiques Garnier (Collection Histoire des techniques)

Geslin, P. (dir.). (2017a). L’anthropotechnologie : Cultures et Conception. London: ISTE Editions (Collection Sciences, Société et Nouvelles Technologies).

Geslin, P. (2017b). 4. La beauté de l’équation. L’anthropologue et l’ingénieur dans les processus de conception. In G. Anichini, F. Carraro, P. Geslin & G. Guille-Escuret (dir.). Technicité versus scientificité : tensions et équivoques. pp161‑183. London : ISTE Editions.

Geslin, P. (1999). L’Apprentissage des mondes. Une anthropologie appliquée aux transferts de technologies. Toulouse: Octarès/ Paris: Éditions de la Maison des Sciences de l’Homme.

Gibson, J.-J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.

Haroche, C. (2008). L’avenir du sensible. Les sens et les sentiments en question. Paris: Presses Universitaires de France.

Hoc, J.-M. (2004). 31. La gestion de situation dynamique. In Pierre Falzon (dir.). Ergonomie. pp. 515‑530. Paris: PUF.

Kruijff, E., Swan, J., & Feiner, S., (2010). Perceptual issues in augmented reality revisited. 9th IEEE International Symposium on Mixed and Augmented Reality 2010: Science and Technology, ISMAR 2010 Proceedings, p. 3-12.

Landragin, F. (2011). De la saillance visuelle à la saillance linguistique. In Olga Inkova (dir.). Saillance. Aspects linguistiques et communicatifs de la mise en évidence dans un texte. pp. 67‑84. Besancon: Presses Universitaires de Franche-Comté, Annales LitteÏraires de l’Université de Franche-Comté.

Le Breton, D. (2013). L’adieu au corps. Paris: Ed. Métailié.

Leplat, J., & Cuny, X. (1977). Introduction à la psychologie du travail. Paris: PUF

Levy, P. (2007). Sur les chemins du virtuel. Université Paris 8. https://manuscritdepot.com/edition/documents-pdf/pierre-levy-le-virtuel_01.pdf

Mériau, O., & Rousseau, T. (2017). Mieux travailler à l’ère du numérique : Définir les enjeux et soutenir l’action. La Revue des Conditions de Travail - ANACT, 6, 6‑13.

Merleau Ponty, M. (1945). Phénoménologie de la perception. Paris: Gallimard (Collection Bibliothèque des idées).

Montuwy, A., Cahour, B., & Dommes, A. (2019). Using Sensory Wearable Devices to Navigate the City: Effectiveness and User Experience in Older Pedestrians. Multimodal Technologies and Interaction, 3(1), 1‑17.

Moreau, G., & Normand, J.-M. (2018). Voir l’invisible : de la vision par ordinateur aux réalités augmentée et Virtuelle. Revue Française d’Histotechnologie, 30 (1), 51‑71.

Pinet, J. (2011). Traitement de situations inattendues d’extrême urgence en vol : test d’un modèle cognitif auprès de pilotes experts. Thèse de doctorat, Psychologie et Ergonomie. Université de Toulouse 2 Le Mirail.

Poizat, G., & Durand, M. (2017). Réinventer le travail et la formation des adultes à l’ère du numérique : état des lieux critique et prospectif. Raisons Educatives, 21, 10‑28.

Sadin, É. (2015). La vie algorithmique. Critique de la raison numérique. Paris: L’Échappée (Collection “Pour en finir avec”).

Salini, D., Sarmiento Jaramillo, J., Goudeaux, A., & Poizat, G. (2018). Profesiones de servicio y digitalización: implicaciones y sugerencias para la concepción de procesos de formación. Laboreal, 14, (2). http://laboreal.up.pt/files/articles/15_30_3.pdf

Serres, M. (2013). Variations sur le corps. Paris: Ed. Le Pommier.

Svétoslavova A., Burkhardt, J.-M., Mégard, C., & Ehanno, P. (2007). L’ergonomie de la réalité augmentée pour l’apprentissage : une revue. Le travail humain 2007/2, (70), 97‑125.

Theureau, J. (2004). Le cours d’action : méthode élémentaire (2e édition). Toulouse: Octarès.

Valenduc, G., & Vendramin, P. (2019). La fin du travail n’est pas pour demain. Notes de prospective, ETUI, The European Trade Union Institute, 6, 1‑16.

Valenduc, G. (2017). Au doigt et à l’œil. Les conditions de travail dans des environnements digitalisés. Hesamag, ETUI, 2017(2), 12‑16.

Varela, F., Thompson, E., & Rosch E. (1993). L’inscription corporelle de l’esprit. Sciences cognitives et expérience humaine. Paris: Ed. Le Seuil. (Collection « La couleur des idées »).

Vermersch, P. (2006). Contrepoint. L’explicitation phénoménologique à partir du point de vue radicalement en première personne. In F. Jeffroy, J. Theureau et Y. Haradji (coord.) Relation entre activité individuelle et activité collective. pp. 129‑146. Toulouse: Octarès.

Wisner, A. (1997). Anthropotechnologie : vers un monde industriel polycentrique. Toulouse: Octarès.

Haut de page

Notes

1 Project funded by the Hasler Foundation (“Cyber-Human Systems” thematic), run by Prof. Sabine Süsstrunk at EPFL (Ecole Polytechnique Fédérale de Lausanne).

2 “Visual fatigue is caused by repeated visual effort (for example, the repetition of major changes in convergence distance) and more often than not combines perceived symptoms with physiological signs.” Arnaldi, B.,Cotin, S., Couture, N., Dautin, J.-L., Gouranton, V., Gruson, F., & Lourdeaux, D. (2018). RA and AR - Myths and Realities. Paris: ISTE Ed. p. 185

3 Visual discomfort is a visual problem immediately perceived by the observer through the integration of one or more negative sensations (for example, ocular pain, irritation, double or blurred vision, convergence difficulties, etc.).” (Ibid.)

4 Fire mode: colorimetric scale from 0-650°, Person mode: 0-60°. Each mode has a “fixed” sub-mode (constant contrast) and an “adaptive” sub-mode (contrast varies in accordance with the thermal environment).

5 Test situations are described in part 4.2.1.

6 To prevent incoming oxygen

7 Instructor, La Rama training centre, November 2016.

8 Backdraft is the burning of superheated gasses that accumulate in an enclosed space and explode when there is an intake of air, at which point it has all the characteristics of an explosion: blast of air, deflagration, etc. (CIS Pont de Chéruy: http://cs.pontdecheruy.free.fr/livres/livre2/2113.html)

9 Bullard Eco X (2.75’ screen, 240x180 px resolution, HFOV: 60°, High: 60°, Low: 40°)

10 Intermediate, 26.06.2019, Neuchâtel (boiler room)

11 Expert, 19.03.2019, Neuchâtel (boiler room)

12 Expert, 19.03.2019, Neuchâtel (boiler room)

13 This point has been demonstrated in numerous works of research, but worthy of particular note is that of Charbonneau (2010), a clinical study of night-vision systems for the aerospace industry: “there is rivalry between the “assisted” eye which sees the image on the screen and the “unassisted” eye which sees the exterior scene. This results in a deterioration of image quality perception. The information extracted from these images can make it difficult to recognise targets and, more generally, reduced visual performance.” (p.58)

14 Novice, 12.03.2019, Neuchâtel (boiler room)

15 Expert, 19.03.2019, Neuchâtel (boiler room)

16 The Moverio screen used during S.A.V.E. and projecting a virtual image at a distance of between 2m and 5m allowed the FFs to maintain a “virtual-free” zone that is sufficiently large to ensure this “ocular exit”.

17 Expert, 27.09.2019, Colombier (fire brigade training camp), outdoor condition

18 “Salience is first and foremost the surfacing of a form from a background, i.e. an element being particularly outstanding to the point of catching attention and offering a hook, a starting point to comprehension.” (Landragin, 2012)

19 Novice, 18.03.2019, Neuchâtel (boiler room)

20 Novice, 18.03.2019, Neuchâtel (boiler room)

21 The order was randomly chosen for each participant in order for the chronology for using each device to have as little influence as possible.

22 Expert, 19.03.2019, Neuchâtel (boiler room)

23 Expert, 27.09.2019, Colombier (fire brigade training camp), white smoke conditions

24 Expert, 30.09.2019, Colombier (fire brigade training camp)

25 This point is mainly based on post-test verbalisation data, but also in part on experience narratives not provided here.

26 During fire interventions FFs are equipped with masks and with bottles containing between 6 and 9 litres of oxygen (i.e. 20 to 40 minutes of oxygen per carrier).

27 Expert, 30.09.2019, Colombier (fire brigade training camp)

28 In this respect our results are similar to observations made with regard to the integration of AR into the field of surgery: surgery is essentially based on visual perception and surgeons are trained to interpret visual inconsistencies as indicators of possible problems or pathology. Thus, in order to make the virtual representations as realistic as possible, each geometric detail or texture must be integrated and cannot be deleted to make the representation lighter, as is done in other fields (e.g. industrial design, architecture). Arnaldi, B.,Cotin, S., Couture, N., Dautin, J.-L., Gouranton, V., Gruson, F., & Lourdeaux, D. (2018). Virtual Reality and Augmented Reality - Myths and Realities. Paris: ISTE Editions. Ch.1 p.43

29 “Spatial representation” refers here to the way in which an individual understands the elements of an environment, along with their absolute and relative positions. (…). Spatial understanding is mainly constructed using two sources: an external source (perception) and an internal source (memory)” (Dutriaux & Gyselinck, 2016, p. 422)

30 “Affordance specifies the actions that may potentially be applied to the elements of an environment, in reference to the body and to the latter’s capacity to act, given the current position of these elements in relation to the body (for example, how easy it is for a given body at a given time to climb stairs or sit in a chair). (Dutriaux & Gyselinck, 2016, p. 423).

31 Firefighter’s verbalisation.

Haut de page

Table des illustrations

Titre Fig. 1. Diagram of the user-centred design methodology and contexts of use applied to S.A.V.E. project.  Fig. 1. Schéma de la méthodologie de conception centrée utilisateurs et contextes d’utilisation appliquée au projet S.A.V.E.
URL http://journals.openedition.org/activites/docannexe/image/5931/img-1.jpg
Fichier image/jpeg, 172k
Titre Fig. 2. “Prototype V1” allowing the choice of optimal configuration for each participant.  Fig. 2. Prototype V1 permettant le choix de configuration optimale pour chaque participant
Légende (on the left, the test bench to test the different configurations; on the right, the different configurations offered)  (à gauche, le banc d’essai pour tester les différentes configurations ; à droite, les différentes configurations proposées)
URL http://journals.openedition.org/activites/docannexe/image/5931/img-2.png
Fichier image/png, 586k
Titre Fig. 3. “Prototype V2” dynamically tested.  Fig. 3. Prototype V2 testé en dynamique
Légende (left and centre: embedded system on helmet and remote control; on the right: the two configurations tested)  (à gauche et centre : système embarqué sur casque et boitier de contrôle ; à droite les deux configurations testées)
URL http://journals.openedition.org/activites/docannexe/image/5931/img-3.jpg
Fichier image/jpeg, 164k
Titre Fig. 4. Diagram of the methodological approach and data collected.  Fig. 4. Schéma de l’approche méthodologique et des données collectées
URL http://journals.openedition.org/activites/docannexe/image/5931/img-4.jpg
Fichier image/jpeg, 208k
Titre Fig. 5. Tools for collecting and analysing gestural and choreographic data.  Fig. 5. Outils de recueil et analyse des données gestuelles et chorégraphiques
Légende (left: path study, right: study of times, gestural and postural notations).  (à gauche : étude des parcours, à droite : étude des temps, notations gestuelles et posturales)
URL http://journals.openedition.org/activites/docannexe/image/5931/img-5.png
Fichier image/png, 545k
Titre Fig. 6. Observations and interviews carried out in training centres.  Fig. 6. Observations et entretiens menés dans les centres de formation
Légende (Switzerland, on the left; England, on the right)(Suisse, à gauche ; Angleterre, à droite)
URL http://journals.openedition.org/activites/docannexe/image/5931/img-6.png
Fichier image/png, 475k
Titre Fig. 7. On the left, mimed gestures of progression within a space; on the right, observation of a fire simulation training session, learning to react to and withstand thermal phenomena.  Fig. 7. À gauche, mime des gestes de progression dans un espace ; à droite, observation d’un entraînement de simulation incendie, apprendre à réagir et supporter les phénomènes thermiques
URL http://journals.openedition.org/activites/docannexe/image/5931/img-7.png
Fichier image/png, 662k
Titre Fig. 8. Postures for viewing thermal information during the reconnaissance phase.  Fig. 8. Postures de visualisation de l’information thermique lors de la phase de reconnaissance
Légende (on the left: gesture simulated during an interview, on the right: thermal visualisation).  (à gauche : geste simulé lors d’un entretien, à droite : visualisation thermique)
URL http://journals.openedition.org/activites/docannexe/image/5931/img-8.png
Fichier image/png, 500k
Titre Fig. 9. Schematic modelling of building fire intervention activity with thermal camera.  Fig. 9. Modélisation schématique de l’activité d’intervention en incendie de bâtiment avec caméra thermique
Légende (adapted from Leplat & Cuny, 1977)(adapté de Leplat & Cuny, 1977)
URL http://journals.openedition.org/activites/docannexe/image/5931/img-9.jpg
Fichier image/jpeg, 132k
Titre Fig. 10. Boiler room, controlled environment for static tests.  Fig. 10. Chaufferie, environnement contrôlé pour les tests en statique
Légende On the left: test room in full light (visible image), at the centre and on the right: obstacles and individual in the field-of-view in semi-darkness (thermal image).  A gauche : salle de test en pleine lumière (image visible), au centre et à droite : obstacles et individu dans le champ visuel en semi-obscurité (image thermique).
URL http://journals.openedition.org/activites/docannexe/image/5931/img-10.jpg
Fichier image/jpeg, 32k
Titre Fig. 11. Military training area and bunker, used for tests in realistic conditions.  Fig. 11. Zone d’entrainement militaire et blockhaus, utilisés pour les tests en conditions réalistes
URL http://journals.openedition.org/activites/docannexe/image/5931/img-11.jpg
Fichier image/jpeg, 36k
Titre Table 1. Summary of the results of the static tests of phase 1 - choice of configuration.  Tableau 1. Synthèse des résultats des tests statiques de la phase 1 – choix de configuration
URL http://journals.openedition.org/activites/docannexe/image/5931/img-12.png
Fichier image/png, 125k
Titre Table 2. Summary of the results of the site reconnaissance tests in phase 1.  Tableau 2. Synthèse des résultats des tests de reconnaissance du lieu de la phase 1
URL http://journals.openedition.org/activites/docannexe/image/5931/img-13.png
Fichier image/png, 98k
Titre Table 3. Summary of the results of the dynamic tests of phase 2.  Tableau 3. Synthèse des résultats des tests dynamiques de la phase 2
URL http://journals.openedition.org/activites/docannexe/image/5931/img-14.png
Fichier image/png, 196k
Titre Fig. 12. Tools for analysing verbalizations, gestures and flow of gestures.  Fig. 12. outils d’analyse des verbalisations, gestes et flux de gestes
URL http://journals.openedition.org/activites/docannexe/image/5931/img-15.png
Fichier image/png, 539k
Titre Fig. 13. Search strategies with thermal camera.  Fig. 13. Stratégies de recherche adoptées avec caméra thermique
Légende (on the left the experts; on the right the intermediates and novices)  (à gauche les experts ; à droite les intermédiaires et novices)
URL http://journals.openedition.org/activites/docannexe/image/5931/img-16.jpg
Fichier image/jpeg, 144k
Titre Fig. 14. Search strategies adopted by both experts and novices with S.A.V.E.  Fig. 14. Stratégies de recherche adoptées à la fois par les experts et les novices avec S.A.V.E.
URL http://journals.openedition.org/activites/docannexe/image/5931/img-17.png
Fichier image/png, 155k
Titre Fig. 15. Notation diagram for the “choreographies” adopted by users, comparison of the use of the thermal camera (CT) and the AR device depending on scenario and expertise.  Fig. 15. Schéma de notation des « chorégraphies » adoptées par les utilisateurs, comparaison de l’usage de la caméra thermique (CT) et du dispositif de RA selon les scénarios et l’expertise.
URL http://journals.openedition.org/activites/docannexe/image/5931/img-18.png
Fichier image/png, 571k
Titre Fig. 16. Body protection with deported image (left) and worn on self (right) (same participant).  Fig. 16. Protection du corps avec image déportée (gauche) et portée sur soi (droite) (même participant)
URL http://journals.openedition.org/activites/docannexe/image/5931/img-19.png
Fichier image/png, 402k
Haut de page

Pour citer cet article

Référence électronique

Carole Baudin et Laura Maillard, « The bodily challenges of augmented vision applied to firefighting activity »Activités [En ligne], 18-1 | 2021, mis en ligne le 15 avril 2021, consulté le 12 mai 2021. URL : http://journals.openedition.org/activites/5931 ; DOI : https://doi.org/10.4000/activites.5931

Haut de page

Auteurs

Carole Baudin

Haute Ecole Arc Ingénierie, HES-SO (Haute Ecole Suisse Spécialisée de Suisse Occidentale), Espace de l’Europe 11, 2000 Neuchâtel (Suisse) ; carole.baudin@he-arc.ch

Laura Maillard

Haute Ecole Arc Ingénierie, HES-SO (Haute Ecole Suisse Spécialisée de Suisse Occidentale), Espace de l’Europe 11, 2000 Neuchâtel (Suisse) ; laura.maillard@he-arc.ch

Haut de page
  • Logo SELF
  • Logo DOAJ - Directory of Open Access Journals
  • OpenEdition Journals
Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search