1Machine learning (ML) techniques confer to computational systems the capacity to build and use rich representations from raw data sets, through specific training procedures. Epistemological questions surrounding this process are currently attracting growing attention from philosophers of science and epistemologists [Grote, Genin et al. 2024]. When human agents (e.g., policy-makers, medical doctors, or scientists) rely on ML algorithms to form beliefs and decisions about a target system, epistemological issues revolving around the notion of justification are inevitably raised. How can such beliefs be justified, given that ML algorithms play a decisive role in their formation? A common default assumption is that statistical models generated by machine learning from data are justified to the extent that they are predictively accurate. Anything that threatens predictive accuracy at the level of model development (e.g., because conditions for accuracy required by statistical learning theory are not met) or model deployment (e.g., because the environment is too unstable) downplays the justification of the beliefs and decisions acquired by ML.
2The focus on predictive accuracy, however, obscures other fundamental epistemological issues surrounding the use of ML systems. First, it is well known that a machine-learning system may be predictively accurate without providing explanations for each prediction or giving users a sense of understanding of what the system is doing. A commonly observed feature of ML techniques is that performance improvements are generally not accompanied by improvements in understanding, and sometimes even lead to greater opacity [Sullivan 2022]. This is particularly the case in Deep Learning (DL), where data are typically encoded into latent spaces of high dimensionality for which no intelligible interpretation is available. This problem, which is now being tackled head-on by the field of Explainable AI, raises fundamental conceptual questions about what it means for a model built by machine learning from data to be understandable at all. Second, whether machine-learning systems provide understanding to their users depends on a larger number of factors, including the technology of data production and collection, the knowledge independently possessed by users about the target domain, and the social arrangements in which technology and knowledge are embedded. For instance, the epistemic value of ML algorithms for the classification of medical imagery cannot be assessed independently from the way medical knowledge and decision-making are socialized through practices such as seeking a second opinion from a peer [Cai, Winter et al. 2019].
- 1 Following common usage [Chollet 2021, 7], we denote by “Shallow Learning models” the complement of (...)
3The goal of the present paper is to suggest that those two blindspots of accuracy-based epistemologies of ML are not entirely independent and may be corrected simultaneously by considering the role of interactivity in the epistemology of ML models. To do so, we propose a case study of the Interactive Machine Learning application CoMo [Matuszewski 2020], allowing users to define sound-gesture mappings, for uses in various contexts such as artistic practice, education, or motor rehabilitation therapy. Although the ML architecture behind CoMo involves only Shallow Learning (SL) models, which typically allow for some limited degree of human interpretability, significant issues of understanding are nonetheless raised by the misalignment between the representations of gestures generated and exploited by motion sensors, digital signal processing algorithms and ML algorithms on the one hand, and the socially constructed, meaningful representations of gestures that human users may form from first-person and third-person perspectives.1
4Our original suggestion, based on the case study, is that an improved understanding of ML models may be obtained primarily by dynamically interacting with them, in addition to merely explaining or interpreting them, as the growing research in Explainable and Interpretable AI often assumes. The paper is structured as follows. In section 3, we introduce the CoMo framework as an instance of the Interactive Machine Learning (IML) paradigm, before focusing, in section 3, on the challenges for understanding raised by the distortions between the user-centric gesture representations and those processed within CoMo. Section 4 explains how patterns of interaction between users and the models induce understanding by progressively aligning human and machine representations of gestures. The question of whether, and to what extent, the resulting account of Interactive Understanding applies to DL architectures is addressed in section 5. We conclude that some SL techniques at least have an enduring epistemological potential, if one cares to look beyond accuracy and take understanding seriously in the epistemology of ML.
5CoMo is a software framework that aims at providing users with IML [Dudley & Kristensson 2018] to design sound/gesture mapping. In a nutshell, the function of CoMo is to enable users to ostensibly define gesture categories and associate them with specific sounds, so that when the system later recognizes instances of that category, it triggers the corresponding sound.
6IML was introduced by Fails & Olsen [2003], several years before the DL wave, in a task where users could improve the performance of a classification algorithm by periodically correcting its predictions, and later formalized by Amershi, Cakmak et al. [2014]. In their review, Dudley & Kristensson [2018] define IML as “distinct from classical machine learning in that human intelligence is applied through iterative teaching and model refinement in a relatively tight loop of ‘set-and-check’”. IML is also related to the concept of “Human-Centered Machine Learning” [Gillies, Fiebrink et al. 2016], which introduces practices and approaches from the field of Human-Machine Interaction into ML and stresses the role of context and user co-adaptation in ML processes.
7The design of CoMo follows these principles, enabling users to intervene periodically in the ML process to define the categories they want the system to learn, check that the gestures they perform are correctly classified, and change their definitions or performance strategies accordingly. More precisely, the use of CoMo alternates between training and performance phases. In the training phases, the user provides the system with instances of a gesture category they want to associate with a given sound, through demonstrations of the gesture recorded by means of motion sensors on a mobile device (typically a smartphone). From this recording, an HMM of the gesture category is constructed, taking processed motion descriptors as parameters. This process may be repeated for each gesture category the user wants to associate with a sound. In the performance phase, the system analyzes in real time the processed data coming from motion sensors and compares them with the model to pick the most likely category given the incoming data, and plays the corresponding sound file. An important challenge is to facilitate the ML procedure so that the system can be trained with very few examples, allowing the user to test the system in a rapid loop of trial and error, as depicted in Fig. 1. It is crucial that users can always return to the training phase to add new instances, remove old ones, or define new categories. In typical uses, the user is expected to move back and forth between the two phases, rather than train the system once and for all, before performing with it.
8The system is entirely based on Web technologies, which makes it accessible from any device running a Web browser, such as smartphones equipped with motion sensors (i.e., accelerometer and gyroscope). The possibility of using smartphones also allows for setting up and distributing the system quickly in a large variety of contexts. Moreover, CoMo was designed to work in collaborative and co-located settings, and is usable by a wide range of users without any knowledge of ML.
9The first prototype of the system was implemented and presented in March 2016, to respond to a use case from the company Orbe, which wanted to recognize whether the user was walking, running, or standing still. This prototype was then further developed during the PhD research project of M. Voillot, intended to create tangible and digital educational tools for early childhood [Voillot, Matuszewski et al. 2024]. The central use case was to create embodied storytelling where teachers could record movements that could be associated with sounds and later be triggered by pupils when “re-enacting” the story. Several other scenarios and use cases also served as test beds, such as workshops with artists, designers, and researchers. From these initial prototypes, the application has been developed further through its use in many different contexts: in artistic settings such as musical creation [Magalhaes, Matuszewski et al. 2020], as an experiment system in scientific settings such as the study of movement group coordination with children learning music (EmoDemos project), or in clinical contexts for post-stroke rehabilitation [Peyre 2022].
Figure 1: This figure presents the overall data flow within CoMo, from production of gesture data, to audio feedback through gesture recognition by the ML system. It also highlights (with dotted arrows) the feedback loop arising from the user’s trial-and-error interaction with the ML model.
10Whether particular applications derived from the CoMo framework actually perform the function they are designed to perform depends on the ability of the ML algorithm to construct an accurate model of the gesture categories as defined by the user. Without such an alignment, the classifications of the system may be systematically misunderstood, and the system will not be usable. In order to get a clear diagnosis of this issue, we need to look into some detail into the way gestures are represented in the CoMo ML pipeline.
11The precise definition of gestures varies between research fields. While it is generally associated with notions of meaning and intentionality, it has been modelled and described differently depending on contexts, such as studies in communication [Kendon 2004], [McNeill 2005], sign languages [Goldin-Meadow & Brentari 2017], human-computer interaction [Carfi & Mastrogiovanni 2021], or performing arts [Godøy & Leman 2009]. In this paper, we call gestures the body movements or postures that the users can perform themselves and classify into meaningful gesture categories. Although it may seem circular, this definition only adopts a user-centric approach to gesture as it takes for granted that users have an antecedent general notion of gesture and antecedent notions of specific gesture categories. Any performable movement that is recognized as an instance of a gesture category by the user will count as a gesture for us. This definition may also seem extremely wide, but it is limited by two important constraints. First, there is a performability constraint: only movements that are performable by a user count as gestures. Second, there is a recognizability constraint: what makes a performed movement an instance of a given gesture category is that the user recognizes the gesture as an instance of that category. What unites all instances of a gesture category is thus ultimately the common meaning that the user ascribes to them. Even though the commonsense notion of gesture links the meaning and identity of gestures with their kinetic profile, it is important to keep in mind that the meaning and identity of gestures are both socially constructed, through culture-mediated conventional associations, and capable of individual idiosyncratic variation.
12This user-centric definition raises a difficulty for the collaborative uses of the framework, where a user defines a gesture category not only for herself but also for other users. In such cases, the performability and recognitional constraints need to be understood distributively across all intended users: gesture categories need to be performable by each intended user and recognizable by all intended users. This means that performability and recognizability involve both first-person and third-person representations of bodily movements. While there is no conceptual difficulty in intending a gesture demonstration to define a category of gestures performable and recognizable by other users, it does place constraints and potential limitations on the collaborative use of CoMo, since the existence of aligned first-person and third-person representations of bodily representations cannot be taken for granted. So there may be conflicting views across a group of users as to whether a particular gesture belongs to a specific gesture category [Fdili Alaoui, Francoise et al. 2017]. In any case, what makes a movement a gesture, and a particular movement an instance of a specific gesture category, ultimately depends on the way users intend those movements to be performed and recognized.
13The ML algorithm used in CoMo [Francoise, Caramiaux et al. 2012, Francoise & Bevilacqua 2018] is designed to classify in real time the incoming data into these user-defined categories. The fact that those categories are flexible and open-ended, because they are freely definable and revisable by the users themselves, is not in itself an issue, as it is precisely the point of ML techniques to deal with flexible and open-ended categories, only defined by examples in the absence of explicit, rule-based definitions. Yet, when one considers the details in the way gesture demonstrations are processed in the CoMo framework, it becomes clear that the representations of gestures in the system are at least highly idealized, and perhaps distorted to such an extent that the representational relation between the encoding of the categories by the model and the gesture categories defined by the user may seem broken.
Figure 2: The four layers of movement and gesture representations defined in CoMo.
- 2 To be precise, the accelerometer device reports a complex raw signal that depends on two components (...)
14As shown in Fig. 2, four layers of gesture representations may be distinguished within the CoMo framework. First of all, motion sensors provide a raw signal, in the form of a time series of vectors taken from the 3D accelerometer (x, y, and z axis) and the gyroscope (yaw, pitch, and roll) of the mobile device, giving instantaneous raw accelerometer data2 and angular velocities along 3 axes, sampled between 50 and 100 Hz. This signal is continuously generated as soon as a CoMo client is connected. Intended as a raw representation of an individual gesture or of a series of gestures, such a sequence already leaves out several features usually taken to be essential to many relevant types of human gestures. First, gestures are typically thought to have temporal boundaries. For this reason, the stream of data points from motion sensors does not yet provide a representation of gestures as such, but at best, a representation of movements underlying gestures that only represent gestures after an appropriate segmentation. Second, in the typical use case where a user only uses a single mobile device that they hold in one hand, or attach to some body part, this stream of values only tracks the motion of a single part of the user’s body, leaving out the rich biomechanical dimensions of performing gestures. For instance, for a user who attaches their device to their wrist, moving their hand upwards with a straight or a broken arm might constitute distinct gestures, but their representation will be treated as equivalent if the accelerations measured at the level of the wrist are sufficiently similar. Third, these specific sensors (chosen in virtue of their wide availability in consumer-grade hardware) systematically neglect possibly important gesture parameters such as the absolute spatial coordinates of the body. This means, for example, that jiggling your hand above your head or below your knees will be represented in the same way by the sensors, despite possible semantic differences.
15The raw signal generated by the sensors thus only provides very basic and partial information about motion. Indeed, the characterization and classification of human motion typically use more refined descriptors, such as joint angles and their dynamics. For this reason, a second layer of Digital Signal Processing (DSP) is necessary for the computation of motion data in real-time. The vector associated with each sample is thus extended with information about 3D linear and compressed acceleration (without gravity), average movement intensity regardless of the direction, orientation, and angular velocities. This representation still takes the form of a stream of values, but these refined descriptors provide a better basis for the identification and recognition of gestures. This being said, the information contained at this level of description supervenes on the information contained in the raw signal. This means that the biases induced by the kind of data captured by motion sensors and their typical use (e.g., one mobile device per user) are not corrected.
16In order to obtain a representation of a gesture from this stream of processed data, a segmentation is required. In the learning phase, the user is responsible for this segmentation when they press “start” and “stop” on the interface, before and after demonstrating an instance of a gesture category to be mapped to a given sound. The system then records the values of the processed data time series between those two temporal boundaries and stores them in a memory slot associated with the sound file. When the user associates several gesture demonstrations with the same word, the system treats them as several instances of the same category. This constitutes the third layer of representation, which aims to represent individual gestures, organized into classes corresponding to user-defined categories. Again, since these categories are defined in terms of biased observations, those biases are inevitably inherited in the representation of gesture instances.
17The task of the ML algorithm is then to learn, for each category, a general description in terms of which segments of the incoming data stream may be classified in the performance phase as instances of specific gesture categories among those defined by the user in the learning phase. In CoMo, this description takes the form of a Hidden Markov Model (HMM): a gesture category is represented by a series of states that are assumed to describe the probability distribution generating the data. This is the fourth layer of representation, aiming to represent gesture categories, as opposed to isolated instances.
18Such models embody the Markov assumption, according to which one only needs to consider the current state to predict the next one. This systematically privileges the representation of local relationships over global patterns and long-term dependencies. However, because gestures are typically rather short events, and because the designers allow the model to use sufficiently many hidden states to model their development, it provides in the case of CoMo a limited distortion of gesture categories. A more problematic bias concerns the way the system ultimately provides classifications of gesture instances in the performance phase. As explained above, the CoMo classification algorithm monitors the processed data and selects, at each point in time, the category that is most likely given the processed data vector. In cases where the user has defined only a small number of categories, and in particular, where no null category corresponds to rest, the system will continuously recognize gestures (and play sounds), even when none are intended by the user. Even if this bias can be corrected in practice by the addition of a null category or processed on the output likelihood, it remains that the mechanism consisting of inferring at each new time sample the best match between observed movements and available categories incurs a bias towards overgenerating classifications, namely classifying movements as gestures (and emitting a sound as a consequence), even when no gesture is intended by the user.
- 3 Concrete examples can be found in the following articles and videos: [Voillot, Matuszewski et al. 2 (...)
19To sum up, attention to the details of the way representations of gesture instances and gesture categories are constructed within the CoMo architecture reveals that it includes considerable amounts of idealization and even distortions that threaten the users’ understanding of the gesture recognition system, and ultimately the reliability of the device3. For instance, if a user’s intended gesture category essentially involves some of the features that are within the blindspots of the motion sensors or of ML model, its intended instantiations during the performance phase may be systematically misclassified.
20A fair assessment of these limitations, however, commands to recall that those distortions result from difficult tradeoffs between representational accuracy and usability, given the motion sensor technology present in easily available mobile devices such as smartphones. Furthermore, the fact that genuine threats to understanding are made possible by the biases induced by such tradeoffs does not mean that understanding is not possible, under the right conditions. To answer this question, we need to get a better grip on the potential reasons for understanding failure in ML systems like CoMo, and more importantly adopt a more dynamic and pragmatic point of view, paying attention to the way users interact with the model over time.
21Following the analysis of the way gestures are processed by the CoMo system, it is natural to explain the difficulty of understanding that users typically experience, at least in part, by the distorting idealizations that this processing involves. According to this hypothesis, users have difficulties understanding how the system works, because the system introduces serious distortions in the way it represents and learns the users’ gestures. Some features of gestures (straight or broken arm) that are difference-makers for users’ categorizations are treated as equivalent (if gestures are recorded from a single mobile phone connected to the forearm). Conversely, some features of gestures that are not difference-makers for users’ categorizations (e.g., movements performed in the short time lapse after the user presses “record” and before they start performing their intended gesture) are represented as difference-makers by the system. According to some accounts of idealization [Strevens 2016], this counts as an idealization failure. The problem is not that the system systematically misrepresents some features of gestures. This is inevitable given the necessary tradeoff between representational accuracy and usability. The problem is that those distortions are not limited to non-difference-makers and therefore obscure the categorization of users’ gestures. For this reason, it may be tempting to attribute the lack of understanding to an idealization failure.
22This view may be reminiscent of the Idealization Failure Hypothesis proposed by Sullivan [2024], according to which “complex or opaque ML models fail to enable understanding of real-world phenomena when there is ML idealization failure” [2024, 1446]. A significant difference, though, is that Sullivan’s Idealization Failure Hypothesis is concerned with the understanding of target phenomena in the world by means of ML models, whereas we are primarily concerned with the understanding of the CoMo model by users. However, part of the reason why users fail to understand how the model works in the case of CoMo is that they fail to understand how it learns a model of target phenomena, since its proper function is to learn on the fly gesture categories. Therefore, the question of model understanding is dependent on the question of target phenomena understanding, in the case of CoMo at least. Whatever diminishes target phenomena understanding also obscures model understanding, to the extent that understanding how the model works involves understanding how it learns about target phenomena.
23Idealization failure, however, cannot be the whole story in the case of CoMo. Given that the distortions caused by the motion sensors are hard-wired into the system and that they propagate to all other representational layers, idealization failure is a massive and unavoidable problem. This view leads to an excessively pessimistic view of the prospects for understanding gestures, unless radical changes are made to the motion-sensing infrastructure of the system. However, the whole CoMo framework was designed to be easily used with commonly available mobile devices equipped with motion sensors. Taking those socio-technical constraints as given, the focus on idealization failure suggests that the possibilities for understanding and reliably using CoMo in practice are severely limited. Experience with the way users actually deal with CoMo shows, on the contrary, that some non-trivial degree of understanding is achievable.
24In order to see this, it is necessary to adopt a more dynamic standpoint, taking into account how the understanding of CoMo by users evolves over time. In holding interdisciplinary workshops4 where users such as dancers were introduced to the framework and had the opportunity to extensively test it, we could witness how participants explore the systems through trial and error, and improve their understanding of the gesture classifications by the system. This process is greatly facilitated by the immediate auditory feedback triggered by the recognition results. While the workshops usually start with explaining basic facts about the types of sensors used in the system, it is only through experimenting with the system that users acquire an embodied and situated knowledge about the recognition procedure. Let us describe in greater detail some striking examples observed on such occasions.
25First, after recording a few gesture categories, users usually check whether the system recognizes the gestures well. Interestingly, dancers tend to spend a long time exploring all the different ways a gesture can be performed, while still leading to the expected recognition results evidenced by the sounds. They might spend time trying to test whether a parameter such as the height of the smartphone is important or not. Similarly, they might be surprised that pointing to the left and to the right in the sagittal plane would not be differentiated. Such explorations provide the participants with a first sense of what is actually captured and what is not, and allow them to get a personal “sense” of the information in the raw and processed data. Interestingly, the users first realize “what’s missing”, such as the insensitivity to height or absolute directionality, rather than the specific characteristics of the motion sensors and descriptors used in the system.
26Second, an interesting challenge for understanding is linked to the way the system handles gesture segmentation and its limits. As described above, the start and end of gestures are defined in the recording using the Start/Stop button. This forces users to pay attention specifically to how they initiate and end their demonstrations, in particular if the movement begins from a state of rest and ends with a return to the initial state of rest. Generally, users tend to assume that the system will neglect the initial and final phases of stillness and only record their gestures when they move. In particular, if they wait too long before starting to move, the model will consider this phase of stillness as essential to the category and will not recognize instances if this initial phase of stillness is not respected. The experience of such difficulties forces users to formalize precisely how their gestures should start and stop, and more generally to identify what needs to be clearly demonstrated for their gestures to be correctly classified.
27Third, a common difficulty has to do with the sharing of gesture categories across users. The recognition is generally efficient when the same person does the recording of the examples and the performance. In this case, all the movement’s idiosyncrasies appear as advantages to perform the recognition, just like one would better recognize handwritten words from examples coming from the same writer. Indeed, difficulties arise when participants try to use the gesture recognizer trained by another person, implicitly testing how the system generalizes and whether it is prone to overfitting. Accuracy then highly depends on the way the gesture categories are transmitted to other users. Verbal descriptions of gestures are largely insufficient since it might be difficult to communicate speed and dynamic movement articulations. Learning through imitation then remains the most practical approach. In particular, the real-time recognition feedback of the CoMo system can facilitate such transmission, and the users can literally use this feedback to learn the gestures and how to make the system work well (such feedback is generally called “knowledge of results” in sensori-motor learning).
28It is worth noting that when the system does not respond to what the user expects, there is often an ambiguity as to whether this is due to a model failure or an imprecision in performing the gesture. Such ambiguity also invites the users to try again and make changes in order to obtain the expected results, which might lead to overwriting all the initial gestures to obtain the expected sounds. Such a human adaptation, imposed by the model limitations, can still lead to a predictable and thus usable system.
29In contrast with the explanation based primarily on idealization failure, the initial difficulties experienced by CoMo users suggest an alternative explanation in terms of alignment failure: users’ understanding is impaired when the categories that are relevant to them are systematically misaligned with the categories that are made salient by the ML pipeline. What drives understanding in the case of CoMo is the constant feedback between the training phase and the performance phase, which gradually helps the two systems of categorization to align. The endemic distortions in the motion-sensing infrastructure do not need to be corrected for a better understanding to be obtained. They only need to be circumvented by focusing on gesture categories that are minimally distorted by those biases, e.g., because their definition is insensitive to the relative location of the body part to which the device is attached, just like the sensors. This alignment process is bi-directional in the sense that the users adjust the categories of gestures they define and their demonstrations to get closer to what the machine can identify, so that in return the classifications of the machine are closer to the user-centric categories. It is essential to adopt this dynamic point of view to see that this alignment process is in fact compatible with constant idealization failure. Over time, the endemic idealization failure is compensated by a gradual, bi-directional process of alignment. As long as this process converges toward robust gesture categories (insofar as the gestures that users expect the machine to recognize are indeed predictably recognized) some understanding of the machine-learning model is obtained.
30The original picture that emerges from the case of CoMo is thus one in which understanding a ML model is a dynamic and interactive achievement, consisting of a continual bi-directional adjustment between the user-defined and ML-constructed categories through situated and embodied interaction. What drives this achievement is the existence of a feedback loop connecting the performance phase to the learning phase. Performance errors can be corrected by immediately updating the model with new demonstrations of gestures. While users may gain some understanding by exploring the behavior of the model during the performance phase alone, the specific type of understanding that CoMo users have gained crucially involves the capacity to intervene back on the training of the model. Without this feedback on the training phases, possibilities for alignment remain limited.
31Generalizing from the case of CoMo, a user may come to understand, or better understand, how a ML model works when there is a bidirectional interaction between the user and the model by which (1) the model is successively retrained with new examples provided by the user, and (2) the user adapts over time their examples to the constrained generalizing capacity of the model, in such a way that the user’s intended categorization of the target phenomena aligns with the classifications of the model. Users come to understand how the system works because the way they think about the target phenomena to be modeled gradually aligns with the generalizations of the model, and they experience this alignment by interacting with the model. When this occurs, we shall say that users gain an interactive understanding of the model they have been using.
32Interactive understanding is an original epistemic achievement that shares many features commonly associated with understanding in the recent philosophical literature [e.g., Hannon 2021]. First, understanding is often described in terms of original mental abilities such as grasping causal or explanatory connections between variables. In the case of interactive understanding, CoMo users may also be described as gradually grasping why some examples are better learned by the model than others. Second, understanding does not reduce to a pure psychological experience (akin to the “aha” effect), but includes a normative component. In the case of interactive understanding, that normative component is tied to the successful use of the ML model. A normatively correct interactive understanding is an interactive understanding that does lead to the successful alignment of user and model categorizations and ultimately to the successful use of the system. Third, understanding, unlike (generic) knowledge, may vary in degrees. This is an essential component of interactive understanding, which develops gradually over time. Finally, understanding is often thought to be connected with new behavioral abilities (e.g., understanding a mathematical proof generates the ability to prove distinct but similar mathematical propositions). Interactive understanding, as exemplified by CoMo, also comes with extended abilities to better record gestures and better trigger sounds with them.
33The interactive understanding of ML models is also distinct both from interpretation and explanation, as those terms are commonly applied to ML models. As a matter of fact, SL models are often taken to be interpretable in the sense that they involve parameters that can be mapped to meaningful features and have been selected by the designers of the ML pipeline precisely because they are known or assumed to be relevant to the task at hand. In the case of CoMo, the hidden states of the HMM models, for instance, are intuitively interpretable as short stages through which the gesture passes over time. Gesture categories may be characterized as transitions between such states modulated by probabilities. For instance, and with considerable simplification, a wave-like gesture will be modeled by sequences of upward and downward motion states. Nevertheless, even with such an interpretation in hand, it remains difficult to guess how to create examples that will be reliably classified by the system. This is in fact due to two decisive factors: the choice of the gesture categories and the precision of gesture performance. Those two factors can vary wildly within and between groups of users, with the use case, and can only be evaluated through first-hand interaction with the model. In other words, having an interpretation of the underlying ML model without interacting with it does not suffice for understanding how ML works in CoMo. Conversely, interactive understanding is achievable from the perspective of users who ignore the technical details of the model and, a fortiori, cannot provide an intuitive meaning to its internal parameters. In that sense, one may interactively understand how the model works without having an interpretation of it.
- 5 We do not mean to take a stance on the question whether understanding in general ultimately reduces (...)
- 6 Embodiment is essential in the case of CoMo because CoMo deals with gestures, but it does not follo (...)
34The epistemic achievement obtained by interactive understanding is not reducible to explanation either, at least not in the way the latter is understood in the field of Explainable AI.5 Even though CoMo users, after having interacted in the right way with the model, may grasp some explanatory connection between features of the gestures and the particular generalization capacities of the model, the grasping is closer to an embodied and situated know-how than to a full-fledged propositional knowledge of the features that are likely responsible for the model’s behavior.6
35It is important to keep in mind that several enabling conditions need to be satisfied for this kind of interactive understanding to arise. First, there must be a feedback loop connecting the inference phase back to the training phase of the model. In other words, it has to be possible to easily and quickly update the model after inference. Second, this feedback loop needs to be accessible from the user’s perspective. The user herself needs to be able to intervene in the re-training of the model, for the right sort of two-way interaction to take place. Third, and as a consequence, the update of the model should not require a lot of data to be significant. In other words, it should be possible to reorient the behavior of the model with a small number of examples. While those conditions are satisfied in the case of CoMo, they place a limitation to the kind of ML models that may come to be interactively understood. This, in particular, may cast doubt over the possibility of gaining an interactive understanding of more complex and data-hungry DL models, to which we now turn.
36The account of interactive understanding we have developed so far was suggested by the example of CoMo, and seems tailored for SL systems. Whether it applies to DL models, including for instance modern Large Language Models (LLMs), is far from obvious. First, to start with LLMs, it is well known that some of their general users, being unaware of the basic principles behind the transformer architecture, tend to develop misleading conceptions about the nature of chatbots (e.g., that they are conscious) and about the character of their relationship with them (e.g., an enduring personal relationship) after interacting with them. Such phenomena may appear as prima facie counterexamples to the interactive understanding account that we have fleshed out in the previous section, since more interaction gives rise to less understanding.
37However, on a second look, it could be argued that such misconceptions originating from interactions with LLMs are not counterexamples, precisely because the required feedback loop between model inference and model training is not instantiated in this case. At the inference phase, the interaction between users and the model is unidirectional, in the sense that the user can only prompt the model and explore how it responds to various inputs, but cannot immediately intervene on its training in return.
- 7 We are grateful to an anonymous reviewer for raising this potential counterexample.
38It might be objected, then, that Reinforcement Learning by Human Feedback (RLHF), an essential part of the training phase of current LLMs, does involve an interaction between humans and the model that affects the training of the model. But the humans, in such cases, are not really users of the model, only informants. So the interaction remains unidirectional, in the reverse direction: the informant reacts to the model’s proposal, but obtains nothing in return. What is missing is a closed loop between those two interactions and the possibility of easy and fluid feedback between them, which is the crucial ingredient that leads to interactive understanding in the case of CoMo. Its absence can therefore explain why the same kind of understanding may not be achieved by interacting with LLMs.7
- 8 We are grateful to the same anonymous reviewer for raising this potential counterexample.
39Another possible objection would be that the interaction between users and the model may be considered bi-directional in the inference phase if one takes seriously the phenomenon of in-context learning (ICL), whereby the models display learning behavior after being given new examples within a prompt [Brown, Mann et al. 2020]. In particular, it has been shown that, for some tasks, in certain conditions at least, ICL is functionally equivalent to performing gradient updates on the same examples [Dai, Sun et al. 2023]. Such results, however, are limited to linear tasks (e.g., linear regressions), and it remains unclear how general the equivalence is. So we cannot safely conclude that in general, or even most of the time, the kind of interaction observed in the real-life deployment of LLMs, gives rise to the level of bi-directional adjustment required for the relevant sort of interactive understanding. To that extent, the misunderstandings induced by interacting with LLMs cannot decisively be taken as counterexamples to our account, at least until more general functional equivalence results are firmly established.8 These considerations raise nonetheless the possibility that interactive understanding may occur locally in DL architectures, to which we now turn.
- 9 For a sense of this diversity in applications related to sound and music, including sound-gesture m (...)
40If the misconceptions instilled by using LLMs are not counterexamples to our account, the question of whether the account applies to DL systems at all becomes more pressing. Given the diversity of ways DL techniques may be used in full-fledged, ready-to-use systems, it is useful to distinguish several cases.9
41First, many ML systems involve complex pipelines where heterogeneous techniques may be used in combination. For instance, latent diffusion models typically integrate transformers within diffusion architectures [Rombach, Blattmann et al. 2022]. Similarly, it may be conceivable that DL and SL may coexist within a single architecture, offering both the high performance of DL and the bidirectional interactivity of SL. In the case of a system like CoMo, one may rely on DL techniques to optimize the algorithms that take care of the DSP part of the pipeline, and leave the gesture learning part to SL algorithms. If the DL algorithm is sufficiently fast at inference time, such a system may support the bidirectional interactivity necessary for the relevant sort of understanding. DL in this kind of scenario would however play at best an auxiliary role. The putative DL-driven DSP algorithm that may be used in such a CoMo-like architecture would remain frozen when it operates within the pipeline, and thus would not play any decisive role in the ML process. The system that users may come to understand interactively remains primarily an SL system, qua ML system, even though it may involve components that were antecedently shaped by DL techniques.
42Second, turning to architectures where DL plays a decisive role in the learning process, such as LLMs, it is common to observe, as we pointed out above, a clear division of labor between the unidirectional adaptation of the model to human informants during the training phase (to the extent that it involves RLHF) and the converse adaptation of the user to the model during the inference phase. This is related to deep socio-technological facts concerning LLMs: high accuracy requires very large datasets, which in turn necessitate long and costly training phases, and do not permit the relevant sort of bi-directional adaptation process that drives interactive understanding in the case of CoMo. Even though data collected from the behavior of users may be collected when a LLM is deployed and then used to update the model, such loops are considerably indirect and cannot be noticed, understood nor exploited by general users. The sort of user-centric interactive understanding that we are interested in cannot occur in this way.
43Third, and more interestingly, the progress of few-shot learning techniques in DL suggests however that it is conceivable to obtain interactive understanding in at least some classes of DL architectures. We already mentioned ICL in LLMs, which, for some tasks at least, seems to allow the bi-directional dynamics required for interactional understanding. To take another example closer to CoMo, Sanchez, Caramiaux et al. [2021] used transfer learning to adapt a pre-trained Deep Neural Network (DNN) for image classification to an interactive learning setting for the classification of users’ drawings along user-defined categories (much in the same way that CoMo classifies users’ gestures along user-defined categories). The transfer learning consists of using the stacked layers of the DNN as an encoder, providing lightweight embeddings from which a second, much smaller DNN (i.e., a simple multilayer perceptron) can be trained efficiently with few examples. In this way, Sanchez, Caramiaux et al. [2021]’s DL pipeline could meet the pre-conditions for interactive understanding. Moreover, they report that users who were ignorant of ML concepts were able, after a few trials, to gain an understanding of the system in this way.
- 10 Other techniques involving Model Agnostic Meta Learning (MAML) have been used recently to achieve f (...)
44It might be objected that small multilayer perceptrons are not so different from SL models. If so, this case would not be so different from the first case, where data taken from the outputs of a frozen DNN are used to train a smaller SL model. In practice, this does not make a big difference, especially from the point of view of users who do not have prior knowledge of ML concepts. In any case, this example shows that techniques of transfer learning are available to help some DL models meet the requirements for interactive understanding.10
45To sum up, while the interactive understanding account may cover some DL systems specifically designed to be able to learn from few data points in real time, it is fair to say that interactive understanding remains out of reach for most DL models. This may seem as a limitation of the interactive understanding account, but also as a new insight into the opacity of large DL models. Large DL models remain opaque, not just because they are not interpretable, or because their outputs are hard to explain, but also because, unlike SL models or DL models capable of few-shot learning, they do not easily lend themselves to the sort of interactive understanding that we have been highlighting in the case of CoMo. This comparison between SL and DL thus helps isolate the conditions under which the epistemic benefits of interactive understanding hold, and says something significant about the epistemic limitations of the large DL models that currently attract most of the attention.
46Making ML models understandable by their users is one of the central epistemological challenges of ML. What the detailed examination of the IML framework of CoMo has taught us is that making ML more explainable or interpretable is not the only way to make progress on this front. Making ML models more interactive is another approach that deserves more attention. At least in some cases, where ML involves small datasets, fast training, and fast inference, users can easily intervene on the training and the evaluation of the ML model. In such cases, understanding can be gained by interacting with models rather than explaining or interpreting them.
47As we saw, there are at the same time significant limitations to the scope of interactive understanding. Our case study focused on a SL system, where the conditions for interactivity are the most favorable. The fact that DL models vastly outperform SL models in accuracy on many tasks with standardized benchmarks does not mean that DL models outperform SL in all epistemically relevant criteria. We have isolated one, interactive understanding, that is more easily obtained by SL models (or small DL models) than by large DL models, for a number of reasons that include the socio-technical context in which the systems are deployed. In the case of CoMo, both the challenges to understanding and the opportunities for interactivity are related to the availability of accelerometers and gyroscopes in everyday objects like smartphones, as well as the fact that it can be run on laptops, and operated in various contexts for a variety of users, such as elementary school pupils, dancers, or post-stroke patients.
- 11 We thank four anonymous reviewers for extremely valuable comments that led to significant improveme (...)
48This should encourage us to reconsider the dominance of accuracy in the evaluation of ML methods. There are undoubtedly good reasons to devote resources to improving DL techniques to solve well-defined problems for which highly accurate solutions are required. As a result, however, the systematic investigation of the epistemic virtues of ML models beyond accuracy largely remains an undone science. There are also good reasons to strive for DL models that are smaller, faster, and require smaller datasets to generalize: one of them is to make them more understandable through interaction. A fortiori, those are also reasons not to abandon SL models with high interactive potential in contexts where they are accurate enough. The widespread view that large DL models made SL obsolete, or that the bigger the better when it comes to DNNs, may only reflect the widespread acceptance of a narrow accuracy-based epistemology of ML. Whenever understanding matters more than high accuracy, Interactive Machine Learning has a lot to offer.11