Navigation – Plan du site

AccueilNuméros12Performances de l'archiveDigital Access as Archival Recons...

Performances de l'archive

Digital Access as Archival Reconstitution: Algorithmic Sampling, Visualization, and the Production of Meaning in Large Moving Image Repositories

L'accès numérique comme reconstitution de l’archive : l'échantillonnage algorithmique, visualisation et la production de sens dans les archives d’images en mouvement numérisées à grande échelle
Eef Masson et Christian G. Olesen


Cet article étudie comment l’analyse et la visualisation de caractéristiques sensorielles de l’image en mouvement pourraient contribuer à une reconstitution des archives audiovisuelles, et comment cela affecte comment les objets qui constituent ces archives acquièrent du sens. À ce but, l’article s’inspire du projet de recherche en cours The Sensory Moving Image Archive (SEMIA). Le point de départ du projet SEMIA était l’observation que les utilisateurs étaient contraints par des pratiques courantes de description archivistique qui donnent accès aux archives d’images en mouvement numérisées. En général les catalogues institutionnels ou les systèmes de gestion de contenu ont tendance à se servir de labels soi-disant ‘sémantiques’ : des mots-clés ou des tags qui facilitent l’identification d’entités qui figurent dans ou qui sont associés aux images en mouvement (par exemple personnages, lieux, dates, genres, etc.). Ces catégories sont produites ou manuellement (avec des résultats souvent fragmentaires) ou — comme c’est le cas de plus en plus souvent — avec des outils qui permettent de créer des métadonnées de manière automatisée automatiquement ou sémi-automatisée. Pourtant, la dépendance de descripteurs sémantiques implique une logique de recherche ciblée qui présuppose que les utilisateurs savent ce dont ils cherchent ; en outre, la recherche est profondément influencée par les cadres d’interprétation qui dirigent la création de catégories et de labels. Cela risque de poser des restrictions à ce que les utilisateurs peuvent trouver ou à comment ils peuvent établir des rapports entre des objets — et finalement aussi à comment ils peuvent utiliser ou réinterpréter le contenu des archives. Le projet SEMIA se propose d’étudier si et comment l’analyse visuelle et la visualisation de rapports sensoriels entre les objets pourraient contribuer à développer une alternative qui peut offrir une forme de navigation davantage explorative.
Notre article propose une réflexion sur ce que cet effort comprend en termes d’attribution de sens dans le contexte des archives d’images en mouvement. Qu’est-ce qu’implique les transformations numériques pour la compréhension des objets et les rapports entre eux ? Comment les conditions de création d’un sens changent-ils à différents points dans le processus de l’analyse computationelle et du design d’interface ? Quels sont les mérites à la fois de la transformation des images et des archives et de nouvelles significations possibles qui se présentent en termes d’accès et réutilisation ? En étudiant ces questions, nous considérons aussi le rôle prospectif de la sérendipité dans la rencontre entre les utilisateurs et les objets de bases de données et de la création de sens, tout en nous penchant sur le prototype de l’outil que l’équipe scientifique de SEMIA est en ce moment en train de développer.

Haut de page

Texte intégral

The authors are indebted to their University of Amsterdam colleagues Floris Paalman (for drawing their attention to O’Connor’s early work) and Nanne van Noord (for helping them understand various technical and conceptual distinctions made in computer vision, and for his elaborations on the history of the field). They would also like to thank Matthias Zeppelzauer (St. Poelten University of Applied Sciences, for providing them insight into the difference between syntactic- and semantic-level feature extraction) and Jim Wraith (Eye Filmmuseum, for sharing his ideas about the profound ‘fungibility’ of binary data). And last but not least, they owe a debt of gratitude to the anonymous peer reviewers of this article, for various editing and literature tips.

1. Prologue

  • 1 At present, O’Connor is a Professor of Information Science at the University of North Texas.

1In 1984, Brian C. O’Connor, a graduate in both Film and Library and Information Studies, successfully defended his PhD thesis, entitled “Access to Film and Video Works: Surrogates for Moving Image Documents”.1 Early on in the text, the author identifies two problems with contemporary tools for access to archival moving images (in practice, institutional catalogues, or the records in them). The first is that said tools “favor the retrieval of documents which respond to needs [that] can be clearly articulated” (O’Connor 1984, p. 2). The second, that those needs have to be expressed in a vocabulary that matches the bibliographic apparatus for “print linguistic documents”: a kind that have dominated document collections for a long time, and which “most procedures for access have been developed” around (Ibid., pp. 2-3). In the case of moving images, O’Connor posits, this vocabulary falls short, because it fails to capture the range of object features that users rely on in interpreting them. In fact, the question is whether such features can be captured in words at all. If the visual is indeed too ‘rich’ or too ‘slippery’ (after Raymond Bellour) for words to adequately describe them (Ibid., pp. 150, 23), then it follows that the features of moving image works simply cannot be satisfactorily queried.

  • 2 In the Netherlands, for instance, archives working with audiovisual collections make use of a contr (...)
  • 3 For example, the Netherlands Institute for Sound and Vision (the country’s main broadcast archive) (...)

2From a contemporary perspective, O’Connor’s observations are striking, because they remind us that despite developments of the last 35 years, our methods and tools for archival access still do not accommodate very well for moving image materials. Even today, users of (institutional) archives rely heavily on written information on the collections they seek access to, often in the form of item-level descriptions. In some cases, this information is no longer stored in separate databases, but integrated into larger collection management systems. Such systems may also contain digitized versions of related still images, or even of the moving image works themselves. However, patrons generally have to approach those works through associated descriptions, that are necessarily verbal. In the past decades, more specialist vocabularies have been developed that are better suited to the needs of various media professionals.2 But the categories for archival description are roughly still the same, and tend to cluster, aside from basic technical information, around data about the works’ production, plot information and information on the people or places featured (often in the form of standardized keywords) or genre (using similarly agreed-upon labels) (see for instance Fairbairn, Pimpinelli & Ross 2016, pp. 5-6). In addition, new technologies for text and speech transcription are beginning to be used to provide denser descriptions of archival objects, allowing users to query collections on the basis of their verbal content.3 Importantly, however, such descriptions still ignore many of the non-verbal qualities of moving images and sound that O’Connor thought our access tools should provide for. For example, they rarely touch upon the visual characteristics of the images (other than to distinguish between black and white, or name the specific colour systems used) or the aural qualities of the works’ soundscape.

  • 4 His formulations here are borrowed from an early report of Intrex, an influential experimental rese (...)

3The other problem O’Connor (1984) mentions, the need for a query centred on “clearly articulated needs”, not only affects those concerned with audiovisual media but impacts on archives and their users more broadly (p. 2 and passim). It presupposes that users are always looking for something specific–while in practice, other strategies for information retrieval, such as browsing, are just as common (Ibid., p. 43). Browsing, for O’Connor, is an approach to “‘discovering pertinent information not known’”, often relying on sources that may “‘not [be] obviously relevant and perhaps not systematically selected’” (Ibid., pp. 45-46). As an archival user, one could adopt such a strategy, among others, in hopes that “one’s professional view might be shaken up or stimulated to an unforeseen insight’” (Ibid., p. 46).4 Formulating a query, a basic principle still in practices of search and retrieve, does not at all align with this approach to discovery. Moreover, the need for a well-defined query arguably restricts users in what they may encounter, which kinds of relations between items are foregrounded for them, and therefore, which kinds of insight about them they can gain.

4Both problems together lead O’Connor to think that in providing access, we should move away from indexing objects and focus instead on creating ‘rich surrogates’. ‘Indexing’ comes down to using terms that (in the classic, semiotic sense) ‘point to’ objects, usually by singling out select features. A ‘surrogate’, in information studies, is understood as something that can deputise for something else, and which, although it results from some kind of a translation – and therefore, like indexing, entails a reduction–shares sufficient attributes with it to be able to stand in for it, in the course of an archival exploration (Ibid., p. 52). For O’Connor, the use of surrogates trumps reliance on series of indices, in that ideally, they represent their referents in richer ways, assuming a broader range of their features. The best kind of surrogate would allow for “the inspection of the collection in a variety of ways” (Ibid., p. 51) and accommodate for “idiosyncracies in user style and needs” (p. 53). Most crucially, however, it would support practices of browsing. One of the key perks of surrogates, for the author, is that they function as instruments of ‘aboutness’ that retain “‘sufficient structure to aid in individual determination of meaning’” (Stephen Robertson, quoted Ibid., p. 136; our emphasis). In other words, compared to the descriptions coming out of standard cataloguing practice, they also help delay the moment in time when such meaning gets assigned.

5The surrogate model O’Connor himself advanced, although it allowed for the inclusion of still images, was largely description-based, and therefore, remained overwhelmingly verbal. This explains perhaps why the thesis betrays a lingering dissatisfaction with the proposed solution. In 1984, when moving image digitization was still a distant promise, the author already fantasized about the prospect of “an algorithm for sampling the document itself” (Ibid., p. 180). In an ideal scenario, then, a surrogate would be something ‘purely’ audiovisual, excerpted from its referent. At the time, O’Connor’s vision was highly speculative. But it in the decades since, he has continued to work on realizing it; for instance, in the course of the 1990s, as part of his exploration of various ‘multimedia’ technologies that became available to film and media scholars (e.g. Augst & O’Connor 1999). Throughout this process, he shunned approaches to representing audiovisual works that were reductive of their potential in terms of meaning, preferring a kind that left the interpretation of data as much as possible to the users, as informed by their specific interests and questions.

2. Introduction

6Today, as ever-larger numbers of archival moving images are becoming digitally available, the prospect of using some sort of a ‘sampling’ technique to make them browsable–as opposed to searchable, through their associated metadata–is becoming more and more feasible. Arguably, a digital version of a film or television programme (itself in fact a sampling, although not in the sense intended by O’Connor) is by its very nature fungible. Bits of it can be substituted or simply taken out. Or otherwise, calculated, measured, and compared with the bits that other items in the same database are made up of. Through comparison, browsing in turn becomes possible. The parameters for it, of course, will depend on the particular transformations applied–allowing in turn for different perspectives on the objects in a database or archive, or the relations between them. Therefore, we would argue, it is surprising that so few in the archival sector have attempted thus far to make digitized moving images browsable by ‘crunching their numbers’ in different ways.

  • 5 The project, now in its final stage, got started in August 2017, and involves partners based at the (...)
  • 6 To include other sensory features, such as auditory ones, would involve very different kinds of tec (...)

7In this article, we want to reflect on our recent experimentation with visual analysis tools for making repositories of digitized moving images browsable. The Sensory Moving Image Archive (SEMIA): Boosting Creative Reuse for Artistic Practice and Research is a two-year, funded research project, involving partners in Film and Heritage Studies and Informatics, the heritage sector, and interactive media design.5 The team’s objective is to establish whether, and if so how, existing algorithms for analyzing and visualizing images-as-data can be repurposed, so as to help shape alternative ways of accessing audiovisual archives. In doing so, we focus (for technical reasons as well as reasons of scope) on colour, shape, texture (in computer vision terms, ‘visual complexity’) and movement (or ‘optical flow’).6 The project is to result in a prototype for a tool that enables users to explore a database through visual object features, rather than to search for items by querying it, using existing labels and categorizations. This way, we seek to delay the moment in time when determinations begin to take place of discrete items’ meanings–a process such labels contribute to–or when the relations between them get interpreted. The tool is designed to deal with large numbers of heterogeneous materials (in terms of production date, genre, but also medium) so as to allow for the revelation of potentially surprising connections among database objects.

  • 7 Considering our focus on colour, shape, texture and movement, the surrogates this produces are prob (...)

8The interface we develop essentially visualizes the relations between discrete objects (that is, fragments from digitized film and television programmes) in our database, on the basis of their shared visual features. In addition, it visualizes the analysis results for single items, which then serve as links to those of other items with like features (or alternatively, highly dissimilar ones). In going this route, we apply at least two types of transformations to the digital objects we begin with. First, we ‘cut them up’ into smaller units (shots), that henceforth serve as the discrete records in our database. And second, we extract specific (visual) information from them, that we subsequently use to compare them, and that also provides the input for the different visualizations. In a sense, the latter procedure comes down to a sampling in O’Connor’s sense: an excerpting of specific information that serves to represent the referent (the shot), reducing it to a select number of features.7 In the process, we basically also transform, or reconstitute, the archive we started off with, and that we seek to provide access to.

9In this piece, we are primarily concerned with the question of what such a reconstitution entails in terms of how archival objects and collections (broadly conceived, as will soon become clear) acquire meaning. Evidently, the abovementioned transformations change the conditions for signification–not only because of the ‘sampling’ of items that takes place, but also in that visual analysis serves to reorient the framework for meaning production along a sensory axis. But how exactly do different digital ‘mutations’ shape those conditions, at various points in the processes of computational analysis and interface design? What are the merits, both of the transformation of images and archives and of the attendant possibilities for new meaning opening up, in terms of archival access and reuse? Specifically, we are keen also to reflect on how well our original intent to delay the moment in time when meaning gets assigned, as well as to place the initiative for it more firmly in the users’ hands, holds up in the course of experimentation.

10In the first section of the article, we elaborate further on the reliance, in practices of archival access, on searching and querying, and the differences in this respect with a more exploratory form of browsing. We discuss the role of labels and categories, and their relation to existing bodies of knowledge (as expressed in natural language). In the next two sections, we then successively deal with some issues that arose, respectively, in the first development phase of the SEMIA project, focused on feature extraction, and in the (partly overlapping) second phase, centred on interface building. Each in their own way, the choices we had to make in these processes cast doubt on the attainability of our initial objective to give more agency to the user in determining the frameworks for assigning meaning. In the last section, we then consider what affordances the browsing of digital archives offers in spite of the aforementioned constraints. In so doing, we try to make productive the notion of serendipitous discovery that figures quite prominently in specialist reflection on information (seeking) behaviour and the organization of information online. We discuss how serendipity–understood here specifically in terms of what Arthur Koestler termed ‘bisociation’–is both built into the environment we are developing, and offered as a possibility to users as they engage with the material our database gives access to.

3. Archival Access: Categorizing, Searching, Browsing

11In a forthcoming article on cultural institutions as data infrastructures, Jussi Parikka (2019) reminds us that the increasingly common practice, in libraries and archives, to replace physical items “with data about items and locations” (relegating the former to storage depots) is in fact “part of the particular information glut since the nineteenth century” (p. 19). The “symbolic and material conditions of access”, he argues, have long since been marked by delivery systems “designated to mediate between the card indexes offering the catalogue order of the holdings” and the actual, tangible objects and their locations (Ibid.). Such systems, designed to support practices of search and retrieval, inevitably rely on prior categorizations (cf. Ibid., p. 15).

12For decades, information scholars have alerted us to the pitfalls of organizing and categorizing (e.g. Bowker & Star 1999) that are key also to practices of indexing, among others in the context of archival description. Ronald Day (2014) points out that in contemporary library and information science, the traditional understanding of a document as ‘evidence’ still dominates (p. 5). But what exactly a document is considered evidence of, depends on particular needs–professional or more informal–and is relative to specific bodies of knowledge and understandings of the world (Ibid., pp. 5-6). In libraries and archives, evidentiary relations are established through cataloguing. This practice relies in turn on taxonomies that bear traces of the particular epistemic contexts in which they emerged. Even regardless of the fact that catalogue labels, for this reason, are oftentimes problematic (for instance, politically suspect), it also means that they do not necessarily match all of an archive’s users’ needs. In querying a system for accessing a repository’s contents, we are always forced to adopt the interpretive frameworks of whoever labelled those contents in the first place. As Day (2014) remarks, this means that “we cannot ask for what is not known”, and therefore, that we are forced to “satisfy our needs by choosing among available options” (pp. 5-6). In the process, we silently agree to ignore the options that are not (obviously) available (even if, hypothetically speaking, those might better match our needs). Arguably, this also stunts us in our ability to discover things not (yet) known.

13Another issue is that in actual practice, users of libraries and archives simply may not know what exactly they are looking for. In the 1960s, information scholars were already aware of different motivations for approaching document collections. While some of those translate easily into queries, others do not (for instance, a vague sense of lack of information, or a wish to monitor a given information environment; see O’Connor 1984, pp. 48-49). In the early 80s, empirical research found that in fact, most users cannot precisely pinpoint their information needs, and therefore, are better served by a system that accommodates browsing (Whitelaw 2015). In the second half of the 90s, scholars of information behaviour, a subfield within information studies, even developed a specialist interest in what they termed ‘information encountering’: the more or less accidental discovery of information (see Foster & Ellis 2014, p. 1022). In media studies, a similar interest emerged in relation to hypertext navigation–a practice often defined in opposition to search because of its more “‘exploratory, discovery-based, serendipitous’” qualities (Bernard Bekavac, as quoted in Neumüller 2001, p. 135). More recently, user research has revealed that in the humanities in particular, search, while ubiquitous, is a highly “imperfect” strategy for the discovery of pertinent sources and information (Whitelaw 2015).

14In a piece on access to digital cultural collections, interface designer and scholar Mitchell Whitelaw (2015) primarily blames this on the fact that search is fundamentally ‘ungenerous’. A search interface, because it demands a query, basically withholds information. It offers only one way in, and therefore fails to “represent the scale and richness of [an online] collection” (Ibid.). Browsing, the “most prominent practical alternative to search” (Ibid.), is very different in this respect. Although the literature on the topic does not provide any precise definitions, there is a level of agreement among authors that browsing is “an iterative process that entails scanning […] or glimpsing […] a field of potential resources, and selecting or sampling items for further investigation and evaluation” (Ibid.). It is more open-ended than search, and may even be performed without a specific goal in mind (or otherwise, said goal may change in the course of an exploration process). In Whitelaw’s view, the model emerging from it challenges the more instrumentalist understanding of human information behaviour in terms of an “input-output exchange” that is common in such disciplines as information retrieval (Ibid.).

15In order to enable browsing at different levels of an information seeking system, interface designers tend to work with so-called ‘information surrogates’: rich, but “compact [and] browsable abstractions of primary content” resulting from a selection and aggregation of features (Ibid.). Selection, in this context, may seem problematic, because it entails that certain features are also filtered out; however, the possibility of browsing inevitably depends on it. For as O’Connor (1984) observes, even if documents are “available by random access[,] the issue of where to look remains” (p. 180). A surrogate, then, is the result of a prior decision as to where the users’ attention should be directed. This decision can either be expressed in words (descriptions) or in (new) images. In the latter case, one option is to somehow ‘sample’ them from the original ones.

4. Feature Extraction: Representation, Semantics, Signification

  • 8 To choose between ‘seeing’ and ‘understanding’, in this context, is a matter of aligning with one o (...)
  • 9 In an AI context, this distinction is generally considered non-essential, as it is ultimately alway (...)

16In the SEMIA project, our main objective was to explore the affordances, in this respect, of technologies developed within the context of computer vision: the branch of informatics concerned with how computers can be taught to ‘see’, or even ‘understand’, (digital) images or images sequences.8 The field, a specialist area within artificial intelligence, it is essentially concerned with developing models for extracting from images key information, so-called visual ‘features’ (a term more narrowly defined here than in what precedes) so that they can be cross-referenced. ‘Extraction’, in this context, involves an analysis: a transformation of said images into descriptions, so-called ‘representations’, that are used in turn to classify them–or rather, elements in them. The idea is that this way, computers automate what humans do, as they recognize elements in a picture or scene.9 Although algorithms for image feature extraction and analysis are generally developed for use in security, military or other commercial applications (Kuhn et al. 2015), they have recently been repurposed to navigate large databases of films and videos; for instance, in the context of digital cinema scholarship (e.g. Kuhn et al. 2013; Flückiger 2019) or for the development of movie recommendation systems (e.g. Bougiatiotis & Giannakopoulos 2018). In SEMIA, we are specifically interested in answering the question whether, and if so how, such techniques point towards a new principle for access to archival collections, complementary to those that inform existing methods and tools.

17At the outset, visual analysis was attractive to us for a number of reasons. First, because it held the promise of compensating for the almost complete lack of information, in the current catalogues and collection management systems, about the visual characteristics of objects in archival moving image repositories. At the very least, extracting visual features and making them productive as means for exploring an archive’s contents might allow for previously unrecoverable items, such as poorly metadated ones, to become approachable or retrievable, and in the process, very simply, ‘visible’. A second, related reason was that such visibility in turn opens up the possibility of new interpretations of the objects or collections at hand. And third, that the use of visual analysis techniques, in alerting users to the potential significance of (perhaps previously unnoticed) sensory aspects of archival moving images, might reorient those interpretations along different axes of meaning. This last prospect sounded particularly promising, especially if the extracted information were to be used also to establish connections or relations between items.

18Taking her cue from Alan Liu, Julia Flanders (2014), has argued that in thinking about how digital collections are constituted, we need to move on from a ‘network’ model, common for analogue museum collections, to more of a ‘patchwork’ model. In doing so, we allow them to transform from aggregations of items with connections between them that “are there to be discovered” to assemblages of things previously unrelated but somehow still “commensurable”, in that they can still be “accommodated to one another” in some way (Ibid., p. 172). The network logic, which also informs traditional cataloguing practices, presupposes that content can be invoked “via a uniform set of parameters from an already known set of available data feeds or fields” (Ibid.). If we were to take visual analysis as a tool for ‘patchworking’ instead, we might be able to connect items “taken from different semantic spaces, from different contexts of purposiveness and original utility” (Ibid.) in order to form new hypotheses about how they might be meaningful in relation to each other, across those existing categories.

  • 10 For instance, in the two examples mentioned, the relevant algorithms assign values within a chosen (...)

19An additional point of appeal in visual analysis was, that even though the relevant techniques, much like the metadating methods of old, involve a labelling of items (in this case, shots, discrete images and/or image sections), the labels involved do not settle to quite the same extent on what said items are supposed to ‘mean’. Arguably, there is a difference, for instance, between identifying a particular film fragment as ‘predominantly green’ or ‘full of movement’, and making the determination that it is part of a larger film belonging to the genre category of ‘comedy’. In the feature extraction process, the visual categories we elected to focus on–colour, shape, texture and movement–do not even translate to such concrete labels (understood in terms of referents shared by most humans) but rather to numerical values in a specific range or on a given scale.10 Within the project context, this only made them more appealing, as it distanced them even further from the categories conventionally used for purposes of archival access.

  • 11 The specific net used was ResNet-101, a convolutional neural network (CNN).

20However, as the extraction work got under way, we began to realize that even algorithms designed to extract such ‘abstract’ features still operate on a presumption, if not of specific meaning, then of a fundamental meaningfullness. For instance, for the purpose of extracting information about shape, the SEMIA informatics team chose to rely on a so-called ‘artificial neural network’ (or ‘neural net’).11 Feature extraction can proceed in roughly two ways: either humans design task-specific algorithms, based on what they think is key information to extract in order for the system to perform specific tasks (for instance, matching similar images), or alternatively, they feed the system lots of data, in hopes that it will teach itself which features are relevant to performing said tasks (either in a more, or in a less supervised way). The latter is a case of machine learning, which essentially involves “systems automatically [learning] programs from data” by generalizing from them (Domingos 2012, p. 2). Neural nets are a type of machine learning devices, and they are often considered extremely powerful algorithms for learning to distinguish patterns (MacKenzie 2017, p. 48). Arguably, those patterns are strongly informed by what humans conceive of as meaningful classes. Given the algorithms’ intended purpose (detection), they primarily serve to identify what computer vision experts call ‘semantic’ entities: people, vehicles, buildings, or other objects. Likewise, neural nets rely on algorithms that assign particular image sections to specific classes–usually classes of ‘things’ (Caesar, Uijlings & Ferrari 2018).

21Yet as it happens, this is precisely the sort of labelling logic we wanted to move away from in the SEMIA project. For this reason, we decided that in order to make the items in our database approachable through their sensory dimensions, we needed to extract features at a lower, more ‘syntactic’ level of our chosen net. Computer vision experts distinguish between different levels of extraction (lower, or more ‘syntactic’, and higher, or more ‘semantic’) based on the complexity of the extracted features. Those range from descriptions relevant to smaller units (such as pixels in discrete images) to larger spatial segments (like sections of such images, or even entire images), whereby the former also serve as building blocks for the latter. By steering for a lower level, we are hoping to select features which allow for matches between images on the basis of similarities or dissimilarities that are less bound to concrete objects than in state-of-the-art computer vision applications.

  • 12 It would lead us too far to discuss in detail how understandings of the terms ‘semantics’ and ‘synt (...)

22However, a key issue with machine learning algorithms, even for informaticians, is that it is difficult to understand how exactly they work (e.g. Domingos 2015, pp. xv-xvi). Therefore, our approach was ultimately a bit of a ‘shot in the dark’. Moreover, as our results show (see fig. 1), it is quite impossible to altogether cancel out semantics (once more in the computer vision sense12) if the nets used are already trained for object recognition. (The second query, for instance, generates a good deal of matches with other pink flowers, some presumably even of the same species.) Historically, it is actually the relation between low-level feature representations and objects that have commonly been exploited in attempts to detect them. Arguably, the focus on such objects is ultimately down to the involvement of humans, because it is people who, at the stages of training or retraining a machine learning device, determine what is or isn’t a valuable analysis result (for instance, an accurate match between two images). Moreover, even if a machine learner is largely ‘unsupervised’, it necessarily embodies “some […] associations beyond the data it’s given” (Domingos 2012), and as Adrian MacKenzie (2017, p. 10) observes, those are often derived from “institutionalized or accepted knowledges”.

Query shots from the SEMIA database, along with four of their ‘nearest neighbours’ (closest visual matches) in the shape, colour, movement, and visual complexity feature spaces.

23A more fundamental issue is that all extraction involves the isolation of specific image features, based on a presumption of their (comparatively greater) meaningfulness. Labelling and classifying, essential to all forms of image analysis, are a matter of making determinations as to what is or is not relevant. This is true even in cases where lower-level features are extracted, and where in semiotic terms (our terminological frame of reference, here, once again being linguistics) no actual matching of a sign to a referent takes place. Any such determination operates according to a logic of reducing, rather than opening up, a wider range of possibilities (MacKenzie 2017, p. 7)–also in terms of what is taken to be visually meaningful. In the case of machine learning, Anja Bechmann and Geoffrey C. Bowker (2019, p. 4) point out, the assumption is often made that classification no longer takes place–because categories are no longer constructed a priori. In reality, however, AI relies on statistics, a science of classification (in their view, just “another word for generalization”, Ibid.). For this reason, it inevitably results in a reduction of possibilities–also, indeed, in terms of which two images can be considered similar, somewhat similar, or entirely dissimilar, and on what basis. From a pragmatic point of view, such reduction is necessary, also in SEMIA; without it, after all, navigating a database would simply be impossible. However, the choices made in the process, even if left in part to a machine, inevitably determine visibility, and thus, delimit which objects or relations the user can consider in terms of their signification.

  • 13 The SEMIA informatics team chose do to colour extraction using histograms in CIELAB colour space (w (...)

24Last but not least, we want to stress that such choices are made not just at the stage of feature extraction, but much earlier on. In the SEMIA project, we not only made prior determinations as to which sensory aspects of the images to focus on; we also had to decide between different approaches to feature extraction. The use of a neural net was one of them–but as it turned out, this method was primarily successful in making matches in terms of shape. For image characteristics such as colour, texture and movement, we had to use task-specific algorithms, which turned out to perform better in those cases. For each of the features mentioned, the choice as to which algorithm to go for entailed a specific perspective on how it could be represented. An illustrative example, here, is the choice between different models for colour representation, which vary among others in terms of whether or not they are designed to match human perception. Inevitably, this also implies certain preconceptions as to what about colour it is, that contributes to a thing’s, image’s, or scene’s meaning.13

5. Interfacing Data: Visualization, Evidence, Experience

25Inevitably, such choices are also made in the visualization of data–the focus of the second, still on-going development phase of the SEMIA project. As mentioned, this phase should result in an interface enabling exploration of moving image fragments on the basis of the labels generated in the feature extraction process. In this next section, we discuss how in this process also, we aim to set the user up for an alternative way of accessing, and ‘making meaning from’, digitized archival materials, as well as the issues we encountered along the way. In light of the project’s current status, we shall do this briefly, and preliminarily. Below, we focus on three principles our interaction design colleagues adhere to as they formulate, amend and implement an interface concept: exploration, generosity and a non-evidentiary relation between data and their visualization. These principles are partly inspired by the project’s objectives, and partly derived from a combination of user preferences, as established both during a workshop organized for the purpose and professional literature on ‘best practices’ in interface design for heritage collections.

26In light of the team’s objective to engender a shift in archival access from search to browse, the choice for an ‘explorable’ interface (understood here as in Marian Dörk, Sheelagh Carpendale and Carey Williamson’s 2011 work on design for ‘information flaneurs’) was an evident one. Since browsing is done not in order to retrieve something specific, but to get a better idea of what a database holds (in hopes that it may trigger an interest, idea, or question), it often happens in the early stages of a project. But it may spring from very different motivations (scholarly, educational, artistic, or other). An explorable interface, then, should consider that users will follow very different paths, and that those paths cannot be determined in advance. The team furthermore took the position that exploration revolves around the relations between objects, which should therefore be central to the visualization of analysis results.

27The second interface principle, generosity, directly derives from Mitchell Whitelaw’s (2015) ideas on the topic. We mentioned the author’s aversion from search interfaces, that withhold information rather than to offer it up in abundance. Generous interfaces, sharing some key features with ‘explorable’ ones, give their users an at-a-glance impression of the collections they mediate, and provide “multiple ways in” (Ibid.). But they also enable closer inspection of individual items via a kind of ‘surrogate’. Unlike the samples produced during our feature extraction process (also abstractions of salient features of the items represented), the ones Whitelaw proposes enable the previewing of specific objects, giving the user a sense of richness at the item level by analogy with that of the collection as a whole. Browsing, in this scenario, entails a toggling between overviews and previews–two key notions in information visualization and interface design (Whitelaw 2015; see also Greene et al. 2000).

  • 14 Note that in the work-in-progress impressions below, the texture feature is not yet included.

28In the envisioned SEMIA interface, overviews are provided in the form of a so-called t-SNE cloud (see Maaten & Hinton 2008) in which each fragment in the database is represented by a key frame (see fig. 2 for an early-stage mock-up). Visualizations revolve around the project’s chosen categories of colour (as in fig. 2), shape, texture and movement, and the cloud can be reconstituted along those four axes.14 Users can preview fragments by zooming in on the cloud (fig. 3) and clicking on them. From within each preview window (fig. 4), they can also access other fragments from the same film (visualized in their temporal relations) and continue to browse from item to item among like fragments if they access the colour, shape, texture or movement analysis area (fig. 5, by clicking the relevant buttons, as seen in fig. 4). In this area, they can also tweak the parameters for analysis. In the movement from overview to preview, the user encounters two types of surrogates, which each in their own way stand in for film or television fragments: first, a key frame, which represents its referent very ‘thinly’, and second, a low-resolution, playable version of the fragment in its entirety–a much ‘thicker’ representation, allowing for up-close inspection.

Figure 2

Figure 2

SEMIA interface prototype opening view: t-SNE cloud visualization of all fragments in the database (here, as clustered by colour).

Figure 3

Figure 3

SEMIA interface prototype: zoom view of the t-SNE cloud visualization of all fragments in the database (as clustered by colour).

Figure 4

Figure 4

SEMIA interface prototype: preview window for a fragment, showing also key frames for other shots from the same film and some associated metadata.

Figure 5

Figure 5

SEMIA interface prototype: colour analysis result for a fragment, showing also key frames of alike shots, and at the top, a colour bar depicting the distribution in terms of dominant colours for the reference fragment.

29As illustrated above, visualization in the SEMIA interface revolves around purely visual relations. In light of the project’s intent to explore other entry points for archival access (in terms of the object features it hinges on), this seems like a consistent choice. However, it arguably also serves to replace old hierarchies of visibility, and therefore, of potential significance, with new ones. Features that are insignificant (because too difficult to identify) in the context of a traditional indexing approach are revalued, and while this entails that previously unrecoverable items may suddenly become visible, others will likely fade into the background. In very large databases, this is inevitable, because here, the desire for generosity always competes with the demands of usability. Some relations will always be foregrounded, to the detriment of others; if not, the view arguably becomes ‘illegible’. As fig. 1 shows, the emphasis in our case is on clear visual matches (particularly on the colour axis) or otherwise obvious connections (as in the nature scenes invoked through queries 1 and 3). The detection of those matches, moreover, is based on a selection of features already deemed salient by the system (the same selection that is used for feature extraction) that in itself amounts to a reduction of the richer kind of surrogate used for previewing. In light of these caveats, we are hard-pressed to maintain our initial hope that visual analysis and visualization might actually serve to delay the moment in time when meaning, or even the scope for it, gets determined. However, a shift undeniably takes place in terms of what it is that makes database objects visible, and in Flanders’ terms, ‘commensurable’. Moreover, as the relations commensurability revolves around are relatively unfamiliar to database users, there is more opportunity here for serendipitous discovery of how items might also be meaningful in relation to each other (as we elaborate in section 6).

30The last, and quite possibly most important principle for interface design–if only because it seems to be the most difficult one to implement–is to convey a non-evidentiary relation between the data interfaced and their visualization (cf. Drucker 2014, pp. 190-191). Within the SEMIA context, data are not valued for what they might help prove about the film and television programmes they stand in for. They serve above all to inspire users to further investigate whatever the visualization suggests might be worth pursuing further (quite possibly in another environment, or even with reference to different sources). In reality, it turns out to be quite hard to communicate such intent via interface design, or to create visualizations that invite users to treat the input data accordingly. In our current assessment, there are two important reasons for this.

  • 15 Manovich uses a type of cloud visualization that is visually similar to our t-SNE cloud, but produc (...)

31The first is that methods and tools for data visualization come with certain expectations as to how they produce meaning. In the study of historical media, specifically film, reliance on image data visualizations is closely entwined with the development of stylometric analysis, where the purpose is to empirically and deductively establish facts about historical film style (for instance in relation to authorship, or to establish developments over time; see Buckland 2008, Olesen 2017). This trend is sustained by such contemporary scholars as Lev Manovich, who try to visualize sensory patterns in large databases of (still) images. In a piece on Cultural Analytics, an approach he pioneered, Manovich (2015) even sees visualization as a tool for establishing a “science of culture”. Of course, such a designation is problematic, as the analyses involved are very much black-boxed and therefore, raise questions as to what exactly they provide evidence of. In addition, the digitized images that large-scale projects depart from tend to be mediated (or remediated) multiple times over–and therefore, cannot serve as a basis for hard claims about the sensory features of the objects they derive from (analogue or digital). Within the SEMIA context, however, such arguments hardly matter, as visualization conventions (including those pertaining to the cloud visualization, which Manovich uses as well) often come with promises of objectivity and transparency, which are very hard to shake.15

  • 16 In response to users’ wishes, also the SEMIA interface prototype provides access to metadata (as he (...)

32A second circumstance that complicates the production of a non-evidentiary interface is the friction such efforts cause between desired functionality and user-friendliness. In interface design, there is a strong tradition of reliance on user research in establishing the requirements for interaction with a system (e.g. Shneiderman et al. 2016, pp. 132-134, 144-147). Such research looks into how respondents commonly approach information or collections and builds on their experience of working with existing interfaces. For obvious reasons, users of archival moving image databases tend to be versed in querying, rather than exploring; moreover, they often rely in the process on semantic descriptors, rather than more abstract sensory labels.16 In the context of a project that seeks to transform access, there is a risk then that such experience will point, precisely, to the sort of interfacing logic it actually seeks to get away from. The question, however, is whether it is at all possible to ignore users’ wishes, because in order for an interface to be minimally intuitive, it does need to take existing interaction habits into account. In other words, our team’s challenge is to find a balance between avoiding the all-too-familiar, so as to ensure sufficient scope for serendipity in the user’s encounter with the objects interfaced, while abiding by certain conventions so as to ensure usability.

6. Closing Reflections: Serendipity and Bisociation in the Exploration of Digitized Moving Image Archives

33Within the context of information studies, the relation between explorative browsing and serendipity is a recurring theme. Even in the early literature, browsing is associated with forms of discovery that are variously characterized as ‘accidental’ or ‘unsystematic’, or that have an element of unpredictability to them (e.g. O’Connor 1984, p. 46). In the oft-cited definition by the writer Horace Walpole (1754), serendipity involves “‘making discoveries by accident and sagacity, of things which one is not in quest of’” (Foster & Ellis 2014, p. 1015). In this last section of our contribution, we want to reflect on how the notion can help us better understand the affordances of a system for the exploration of a moving image database according to the principles we previously discussed.

34Since around the 1930s, the term ‘serendipity’ has often been used, especially in a (social) science context, to characterize discoveries that engender new ways of thinking about, or explaining, a given problem (Andel 1994, pp. 633-634; Foster & Ellis 2014, p. 1015). Key to such understandings of the notion is an element of chance–but not chance in the sense of pure luck. Serendipity is seen, on the one hand, as the product of an unexpected encounter with a fact, piece of evidence or datum obtained fortuitously. On the other, it also involves critical discernment and evaluation (‘sagacity’), specifically of an individual predisposed to appreciate the value of what was unexpectedly stumbled upon, and in relation to pre-existing problems or questions–for as the microbiologist Louis Pasteur once put it: “‘chance favors only prepared minds’” (Andel 1994, p. 635).

  • 17 Arguably, this is also the reason why, as Nanna Bonde Thylstrup (2018) justly remarks, it is so rea (...)

35In recent years, the notion has become quite common also in reflections on how knowledge is acquired in such disciplines as information studies and media studies. But despite its increased popularity, it remains an elusive concept.17 One reason for this is that it is difficult to study phenomena of serendipitous discovery empirically–because it is by definition hard to foresee when they will occur. Another obstacle to both observing the phenomenon and operationalizing the term, is that researchers may not actually want to admit that their discoveries are partly ‘accidental’, for “‘modern scientific endeavour is based on a view of science as something that is totally under control’” (Juan M. Campanario, quoted in Foster & Ellis 2014, p. 1019). In spite of this, attempts have been made to establish where in the information seeking or encountering process serendipity is experienced, among others on the basis of qualitative interviews with different user groups (see for instance Foster & Ellis 2014). This has not led to a single, encompassing characterization of serendipitous discovery, but it did result in more fine-grained understandings of its many forms, and brought attention to its fundamental complexity.

36Arguably, several of our development choices in SEMIA nurture a form of serendipity in the process of accessing, and making meaning from, digitized archives. To demonstrate this, we draw on journalist and writer Arthur Koestler’s interpretation of the phenomenon, which revolves in turn around the notion of ‘bisociation’. Koestler’s 1964 book The Act of Creation, in which he developed the concept, is an attempt at an encompassing theory of creativity, covering its role in science and art but also different forms of humorous communication. Based on uses of the term to date, bisociation can be concisely defined as “a surprising association between disparate previously unconnected pieces of information” (Foster & David Ellis 2014, p. 1026). Key to Koestler’s understanding of the term, however, is that this association, on the face of it, may not be at all logical. In fact, the particular logic of bisociation is akin to that of dreams, in that “at the decisive step of discovery the codes of disciplined reasoning are suspended” (Koestler 1964, p. 178). An often-used, telling example in this context is that of the humorous pun, whereby “two strings of thought [are] tied together by an acoustic knot” (Ibid., p. 65)–notwithstanding the seemingly incompatible contexts and situations they are taken from.

37For our purpose here, it is important to note that Koestler not only considers linguistic examples, but also visual ones, discussing among others so-called ‘optical puns’ (Koestler 1964, p. 182). Play, here, does not centre on sameness in appearance (despite difference in meaning) but involves much vaguer, less defined formal similarities. Examples proffered by Koestler are those of “the caricaturist who equates a nose with a cucumber”, or author and filmmaker Jean Cocteau’s drawing, during a drug rehabilitation period, of “human figures constructed out of long, thin stalks of opium pipes” (Ibid.). Within the SEMIA context, those invoke associations, for instance, with the neural net’s suggestion that the petals of an orchid, when seen in movement, are akin to those of a man gesticulating while speaking (as shown in the third nearest neighbour visualization in fig. 1).

38In the years since Koestler’s publication, the notion has also found footing in cognitive science. Mark Turner and Gilles Fauconnier, as part of their theory of ‘conceptual blending’, established that bisociation is hardly an exceptional phenomenon, and in fact, a fundamental characteristic of human meaning-making and “ubiquitous in everyday thought and language” (Fauconnier & Turner 2002, p. 37). However, as Koestler stresses, it can only ever amount to a surprising, Eureka-like discovery for a person predisposed to doing so on the basis of prior knowledge. Moreover, if bisociation is to serve as a means to profoundly reconsider said knowledge–for instance by integrating previously incompatible lines of thought from different realms (or ‘matrices’) or to explore their interconnections more systematically (Ibid., pp. 212 and 223)–it requires both an open mind a good deal of intellectual curiosity.

39For all those reasons, experiences of serendipity are necessarily elusive–and therefore, we cannot aspire here to predicting when and where they will occur, in users’ interactions with our interface or the analyses they mediate. Ultimately, we can only encourage others to try our prototype out, in hopes that it will elicit reflections even unforeseen by us. Yet in spite of this, the concept of bisociation does invite further reflection on how, within the SEMIA project, we set the user up for serendipitous encounters with the material at hand. In what follows, we try to do this more or less systematically, reconsidering once again the choices we made in feature extraction, respectively visualization.

  • 18 A well-known, early example of media scholars deviating from this logic is the Austria-based “Digit (...)

40A first point to touch upon is our decision, at the very outset of the project, to depart not from an existing archival (sub-)collection, but to compose our corpus out of highly heterogeneous materials. The former option is actually more common, in both informatics and the digital humanities, as it allows for the training of algorithms that are adept at ‘correctly’ identifying semantic entities.18 By electing to combine film and television materials, but also items produced in different eras and parts of the world and for different user contexts (e.g. theatrical and non-theatrical), we brought together materials that tend to be considered apart from each other in media (historical) scholarship. In having the system suggest visual relations between them, we set our users up to at the very least ask the question whether those other connections might be worth considering as well. (Obviously, such an approach would not work, if one’s goal were to make users re-engage with an existing, institutional archival collection–but in light of our investigation of the affordances of feature extraction and analysis, it made sense to push the system to its limits in this respect.)

41A second relevant step we took, this time in the feature analysis process, was to deliberately avoid a logic of object detection, extracting features at a more syntactic level of analysis than is common for this purpose. In the process, we basically chose to ignore, or ‘deactivate’, some of the knowledge the algorithms we worked with had either been fed (by the humans developing them) or acquired semi-independently themselves (in the case of our neural net). In doing so, we sought to enable users to make a type of connections that the researchers involved in building those architectures, likely would not have considered productive. In Koestler’s (1964) words, this comes down to a challenging of “the codes of disciplined reasoning” that also govern practices of information engineering (p. 178). In light of our own objectives, the main promise of this intervention lies in facilitating a comparison of shots which, for informed users, make more sense bisociatively than in a conventional, logical way. We are thinking here among others of matches that may not be alike in the semantic sense, but that still share formal similarities–perhaps even in the manner of optical puns. (We provide an example of this below.)

42Last but not least, some of our present considerations at the interface design stage also serve to afford more serendipity in the user’s encounter with archival moving images. As we previously explained, the current SEMIA prototype opens with a cloud view of clustered shots, which is zoomable (see fig. 2, 3 above). The rationale here is that the user, when beginning an exploration, selects a fragment from the cloud that subsequently serves as a starting point for comparison, and after that, potentially also for the discovery of patterns within a larger batch of items.

43To begin with the initial stage of this process, an advantage of the above approach is that it does not require a directed choice in the form of a query-response interaction with the database. This entails that even if the user is primarily interested in a particular type of shot (for instance, in browsing along the colour axis: one that is predominantly beige), they still need to approach the item in a more roundabout way than by describing their requirements in precise terms. Arguably, this makes for a more chance-based approach, even at the very early stage of item selection, than in classic search–also in a user predisposed to such information-seeking behaviour. In the next step, then, clicking on a given fragment opens a preview window (fig. 4). At this stage, the user may be surprised once more, as the fragment in its entirety may have rather different hues than anticipated on the basis of the key frame selected. (For humans, indeed, it is impossible to predict how a computer system will ‘see’ beige and what sort of definition of ‘predominance’ the matches are based on; the second set of nearest neighbours in fig. 1 is a case in point.)

44In the second phase of the interaction with the database, the user then proceeds from an exploration of intratextual sensory relations, to relations between items in the entire database. At this point, there is scope for a more truly bisociative probing of the relations between them. In the ‘analysis area’ for a given fragment (fig. 5), the user can navigate between alike shots along the different sensory axes, changing selection outputs either by clicking on new query images or by changing the parameters for analysis (note the cursor in the same image). Here, it both becomes more obvious how difficult determinants for likeness are to make out, and what surprising connections this may yield. At this stage of the SEMIA project, our interface does not yet have data behind it; therefore, we invite the reader to briefly consider another example instead. It is taken from the work of Geert Mul, a data artist the project team has been inspired by and worked with on several occasions.

Figure 6

Figure 6

Still from the video portion of Geert Mul, Match of the Day (2004-ongoing).

45Figure 6 shows a still from Match of the Day (2004-ongoing), a video installation that makes use of similar principles for feature extraction and analysis to the ones we rely on in the project. The installation re-appropriates images from satellite television stations around the world, matching them on the basis of visual likeness using image recognition software. In deciding which matches to include in the work, Mul previewed the results the system generated, but only at the ‘mid-range’ level, where the paired images are similar but certainly not identical, and “also different enough that they start to ‘resonate’” (see Mul & Masson 2018, p. 176). Resemblance, in those cases, is generally not a matter of (pure) semantics; moreover, viewers often appreciate the so-called ‘resonance’ between them in much the same way as the sort of optical puns that Koestler spoke of (sometimes even laughing, at times uncomfortably, as they watch the installation). Closer inspection of the work suggests that part of its power derives from the fact that it brings together images, and attendant ideas, from different social, cultural or political contexts, and oftentimes, altogether different realms of thought (Koestler’s ‘matrices’). For instance, a recurrent type of match is that between images taken from reports on fashion or entertainment, and war, that show subtle yet clearly visible similarities (as in the picture above). Arguably, the repeated inclusion of such pairs brings to the fore otherwise hidden patterns, that in turn invite the viewer to reconsider both types of events.

46A last point we want to mention is that within the SEMIA prototype as we currently envision it, we will be adding functionality to create and save one’s own moving image ‘collections’. This should afford the making of other surprising connections, as a starting point for exploring the mutual relations between images serendipitously but also in highly personal ways. Moreover, tools for tagging and bookmarking will empower users to express how they attribute meaning to said connections, and perhaps even encourage others to amend or add onto their interpretations. This way, they should have a hand in determining for themselves which shape their experience of serendipitous discovery takes, and how they act upon or tweak it in the course of their explorations.

47In adding these functionalities, we hope to be able to answer O’Connor’s (1984) initial call, reiterated since the mid-80s, to not only facilitate a surrogate-based form of browsing, but one accommodating for “idiosyncracies in user style and needs” (p. 53). In the process, of course, we are forced to reckon with various limitations, most notably in trying to delay the moment in time when the scope for assigning meaning to what one encounters in a database is determined, or reduced. This is at least partly due to the choices we make (for instance, the decision to radically opt for sensory features) but partly also to the knowledges built into the technologies we rely on to realize our project. Even so, the tool as we envision it will offer various opportunities to explore different paths towards assigning such meaning–and in the process, perhaps even mirror the against-the-grain attitude we have tried to pursue, in repurposing state-of-the art analysis techniques.

Haut de page


Andel, Pek van (1994), “Anatomy of the Unsought Finding. Serendipity: Origin, History, Domains, Traditions, Appearances, Patterns and Programmability”, The British Journal for the Philosophy of Science, 2, pp. 631-648.

Anderson, Richard L. & O’Connor, Brian C. (2016), “What Makes a Movie”, Proceedings from the Document Academy, 1, pp. 1-10. Available from: (accessed 30 April 2019).

Augst, Bertrand & O’Connor, Brian C. (1999), “No Longer a Shot in the Dark: Engineering a Robust Environment for Film Study”, Computers and the Humanities, 4, pp. 345-363.

Bechmann, Anja & Bowker, Geoffrey C. (2019), “Unsupervised by Any Other Name: Hidden Layers of Knowledge Production in Artificial Intelligence on Social Media”, Big Data and Society, advance online publication, pp. 1-11. Doi: 10.1177/2053951718819569.

Bougiatiotis, Konstantinos & Giannakopoulos, Theodoros (2018), “Enhanced Movie Content Similarity Based on Textual, Auditory and Visual Information”, Expert Systems With Applications, 96, pp. 86-102. Doi: 10.1016/j.eswa.2017.11.050.

Bowker, Geoffrey C. & Star, Susan Leigh (1999), Sorting Things Out: Classification and Its Consequences, Cambridge, MA: MIT Press.

Buckland, Warren (2008), “What Does the Statistical Style Analysis of Film Involve?”, Literary and Linguistic Computing, 2, pp. 219-230. Doi: 10.1093/llc/fqm046.

Caesar, Holger, Uijlings, Jasper & Ferrari, Vittorio (2018), “COCO-Stuff: Thing and Stuff Classes in Context”, in 2018 IEEE conference on Computer Vision and Pattern Recognition, n.p.: Computer Vision Foundation, 2018. Available from: (accessed 2 October 2019).

Colque, Rensso Victor Hugo Mora, Caetano, Carlos, Andrade, Matheus Toledo Lustosa De & Schwartz, William Robson (2017), “Histograms of Optical Flow Orientation and Magnitude and Entropy to Detect Anomalous Events in Videos”, IEEE Transactions on Circuits and Systems for Video Technology, 3, pp. 673-682. Doi: 10.1109/TCSVT.2016.2637778.

Day, Ronald (2014), Indexing it All: The Subject in the Age of Documentation, Information, and Data, Cambridge, MA: MIT Press.

Domingos, Pedro (2012), “A Few Useful Things to Know about Machine Learning”, Communications of the ACM, 10, pp. 78-87.

Domingos, Pedro (2015), The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World, New York: Basic Books.

Dörk, Marian, Carpendale, Sheelagh & Williamson, Carey (2011), “The Information Flaneur: A Fresh Look at Information Seeking”, in CHI 2011: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY: ACM, 2011. Doi: 10.1145/1978942.1979124.

Drucker, Johanna (2014), Graphesis: Visual Forms of Knowledge Production, Cambridge, MA: Harvard University Press.

Fairbairn, Natasha, Pimpinelli, Maria Assunta & Ross, Thelma (2016), “The FIAF Moving Image Cataloguing Manual”, unpublished manual, FIAF, Brussels. Available from: (accessed 8 April 2019).

Fauconnier, Gilles & Turner, Mark (2002), The Way We Think: Conceptual Blending and the Mind’s Hidden Complexities, New York: Basic Books.

Flanders, Julia (2014), “Rethinking Collections”, in Longley Arthur & Bode (eds.), Advancing Digital Humanities: Research, Methods, Theories, Houndmills: Palgrave Macmillan, 2014, pp. 163-174.

Flückiger, Barbara (2019), “Avancierte Methoden der computer-gestützten ästhetischen Filmanalyse”, in Sahle (ed.), DHd 2019 Digital Humanities: multimedial & multimodal – Konferenzabstracts, n.p.: Verband Digital Humanities im deutschsprachigen Raum e.V., 2019, pp. 13-21. Doi: 10.5281/zenodo.2596095.

Foster, Allen E. & Ellis, David (2014), “Serendipity and Its Study”, Journal of Documentation, 6, pp. 1015-138. Doi: 10.1108/00220410310472518.

Greene, Stephan, Marchionini, Gary, Plaisant, Catherine & Shneiderman, Ben (2000), “Previews and Overviews in Digital Libraries: Designing Surrogates to Support Visual Information Seeking”, Journal of the American Society for Information Science, 4, pp. 380-393. Doi:<380::AID-ASI7>3.0.CO;2-5.

Koestler, Arthur (1964), The Act of Creation, London: Hutchinson.

Kuhn, Virginia, Craig, Alan, Franklin, Kevin, Simeone, Michael, Arora, Ritu, Bock, Dave & Marini, Luigi (2013), “Large Scale Video Analytics: On-demand, Iterative Inquiry for Moving Image Research”, in 2012 IEEE 8th International Conference on E-Science, n.p.: Institute of Electrical and Electronics Engineers, 2013. Doi: 10.1109/eScience.2012.6404446.

Kuhn, Virginia, Craig, Alan, Simeone, Michael, Puthanveetil, Sandeep & Marini, Luigi (2015), “The VAT: Enhanced Video Analysis”, in Proceedings of the 2015 XSEDE Conference, n.p.: XSEDE, 2015. Doi: 10.1145/2792745.2792756.

Maaten, Laurens van der & Hinton, Geoffrey (2008), “Visualizing High-Dimensional Data Using t-SNE”, Journal of Machine Learning Research, 9, pp. 2579-2605.

MacKenzie, Adrian (2017), Machine Learners: Archaeology of a Data Practice, Cambridge, MA: MIT Press.

Manovich, Lev (2015), “The Science of Culture? Social Computing, Digital Humanities, and Cultural Analytics,” Lev Manovich home page, (accessed 12 July 2018).

Mitrović, Dalibor & Zeppelzauer, Matthias (2011), Syntactic and Semantic Concepts in Audio-Visual Media, PhD dissertation, Vienna University of Technology. Available from: (accessed 29 April 2019).

Mul, Geert & Masson, Eef (2018), “Data-Based Art, Algorithmic Poetry: Geert Mul in conversation with Eef Masson” TMG - Journal for Media History, 2, pp. 170-186. Available from: (accessed 29 April 2019).

Neumüller, Moritz (2001), Hypertext Semiotics in the Commercialized Internet, PhD dissertation, Vienna University of Economics and Business. Available from: (accessed 2 October 2019).

O’connor, Brian C. (1984), Access to Film and Video Works: Surrogates for Moving Image Documents, PhD dissertation, University of California, Berkeley. Available from: (accessed 10 April 2019).

Olesen, Christian Gosvig (2017), “Towards a ‘Humanistic Cinemetrics?”, in Schäfer & van Es (eds.), The Datafied Society: Studying Culture through Data, Amsterdam: Amsterdam University Press, 2017, pp. 39-54.

Parikka, Jussi (2019), “Situated Data: On Art Methods, Cultural Institutions and Infrastructure”, MediArXiv, 22 February 2019, doi: 10.31219/ (Chapter draft for Gabrillo & Zetter (eds.), Articulating Media: Language and Location at the Emergence of the Digital, forthcoming.)

Rosenholtz, Ruth, Li, Yuanzhen & Nakano, Lisa (2007), “Measuring Visual Clutter”, Journal of vision, 17, pp. 1-22. Doi: 10.1167/7.2.17.

Shneiderman, Ben, Plaisant, Catherine, Cohen, Maxine, Jacobs, Steven, Elmqvist, Niklas & Diakopoulos, Nicholas (2016), Designing the User Interface: Strategies for Effective Human-Computer Interaction, 6th ed., Boston, MA: Pearson.

Thylstrup, Nanna Bonde (2018), The Politics of Mass Digitization, Cambridge, MA: MIT Press.

Whitelaw, Mitchell (2015), “Generous Interfaces for Digital Cultural Collections”, Digital Humanities Quarterly, 9. Available from: (accessed 8 April 2019).

Haut de page


1 At present, O’Connor is a Professor of Information Science at the University of North Texas.

2 In the Netherlands, for instance, archives working with audiovisual collections make use of a controlled vocabulary, developed collaboratively since 1993: Gemeenschappelijke Thesaurus Audiovisuele Archieven (GTAA, Common Thesaurus for Audiovisual Archives). See (accessed 13 September 2019). (Thanks to Erwin Verbruggen and Evelien Wolda at Netherlands Institute for Sound and Vision for helping us date this effort.)

3 For example, the Netherlands Institute for Sound and Vision (the country’s main broadcast archive) has recently been experimenting with the use of automated speech recognition to complement its archival description; see for instance

4 His formulations here are borrowed from an early report of Intrex, an influential experimental research project on information sharing and retrieval, initiated by MIT in 1965.

5 The project, now in its final stage, got started in August 2017, and involves partners based at the University of Amsterdam, Eye Filmmuseum (the Dutch centre for cinematography), Netherlands Institute for Sound and Vision (the country’s main broadcast archive) and Studio Louter (producer of multimedia museum presentations). Funding was obtained from the Netherlands Organisation for Scientific Research (NWO).

6 To include other sensory features, such as auditory ones, would involve very different kinds of techniques for data representation and comparison, and is therefore beyond the projects’ scope.

7 Considering our focus on colour, shape, texture and movement, the surrogates this produces are probably not as ‘rich’ as O’Connor might have hoped; however, we very much see this effort as a first step, and the samplings, as complementary to other representations of the objects in a database. Worth mentioning here also is that O’Connor in fact opposes the idea of the shot as a unit of analysis (see Anderson & O’Connor 2016, p. 3). However, a key difference between his endeavour and ours is, that his was inspired much more by a tradition of analysing the (single) film as ‘text’–while ours is ultimately to create different entry points for exploring archival collections, also in terms of the relations between items, as fragments.

8 To choose between ‘seeing’ and ‘understanding’, in this context, is a matter of aligning with one of two broader sets of views on whether what computers are taught to imitate or emulate, is or is not a function of cognition (as opposed to perception).

9 In an AI context, this distinction is generally considered non-essential, as it is ultimately always the ‘real’ world that serves as a touch-stone for accurate identification (which basically entails that the role of mediation is glossed over).

10 For instance, in the two examples mentioned, the relevant algorithms assign values within a chosen colour space (CIELAB), respectively make measurements of the orientation and magnitude of the ‘flow’ within certain regions of an image (see note 13 for more specifics).

11 The specific net used was ResNet-101, a convolutional neural network (CNN).

12 It would lead us too far to discuss in detail how understandings of the terms ‘semantics’ and ‘syntax’ vary between informatics and linguistics, but obviously, the conception of the relation between them differs between both fields–at least in this particular instance of usage. (In discussions of how programming languages work, in contrast, there generally seems to be more alignment in interpretation.)

13 The SEMIA informatics team chose do to colour extraction using histograms in CIELAB colour space (with 16 bins for the description of each colour dimension, resulting in a feature representation of 4096 dimensions), capturing the values used in moving images invariant of their spatial position. Texture, in turn, is a measure of how much clutter there is in a visual scene; for extracting information of this kind, Subband Entropy (see Rosenholtz, Li & Nakano 2007) is used, which expresses a scene’s visual complexity as a single scalar value. For the extraction of movement information, the team measure optical flow by constructing a histogram for which they separately bin angle and magnitude for each cell in a three by three grid of non-overlapping spatial regions (akin to the HOFM approach described in Colque et al. 2017).

14 Note that in the work-in-progress impressions below, the texture feature is not yet included.

15 Manovich uses a type of cloud visualization that is visually similar to our t-SNE cloud, but produced with another application, ImagePlot (based on the open-source image-analysis programme ImageJ, originally developed for the purpose of medical research).

16 In response to users’ wishes, also the SEMIA interface prototype provides access to metadata (as held by our different partner institutions, so in other words: fragmentary and heterogeneous; see fig. 4). However, it does not show them ‘up front’, and purposefully builds in extra steps for those keen to use them to filter analysis results.

17 Arguably, this is also the reason why, as Nanna Bonde Thylstrup (2018) justly remarks, it is so readily used for strategic purposes, for instance in the justification and ‘selling’ of mass digitization projects (pp. 121-26).

18 A well-known, early example of media scholars deviating from this logic is the Austria-based “Digital Formalism: The Vienna Vertov Collection” project (2007-2010). While still focused on a relatively homogenous corpus, it also sought to reveal non-semantic connections between image elements (see Mitrović & Zeppelzauer 2011).

Haut de page

Table des illustrations

Titre Figure 1
Fichier image/png, 651k
Fichier image/png, 397k
Fichier image/png, 402k
Légende Query shots from the SEMIA database, along with four of their ‘nearest neighbours’ (closest visual matches) in the shape, colour, movement, and visual complexity feature spaces.
Fichier image/png, 291k
Titre Figure 2
Légende SEMIA interface prototype opening view: t-SNE cloud visualization of all fragments in the database (here, as clustered by colour).
Fichier image/jpeg, 163k
Titre Figure 3
Légende SEMIA interface prototype: zoom view of the t-SNE cloud visualization of all fragments in the database (as clustered by colour).
Fichier image/jpeg, 232k
Titre Figure 4
Légende SEMIA interface prototype: preview window for a fragment, showing also key frames for other shots from the same film and some associated metadata.
Fichier image/jpeg, 183k
Titre Figure 5
Légende SEMIA interface prototype: colour analysis result for a fragment, showing also key frames of alike shots, and at the top, a colour bar depicting the distribution in terms of dominant colours for the reference fragment.
Fichier image/jpeg, 116k
Titre Figure 6
Légende Still from the video portion of Geert Mul, Match of the Day (2004-ongoing).
Fichier image/jpeg, 85k
Haut de page

Pour citer cet article

Référence électronique

Eef Masson et Christian G. Olesen, « Digital Access as Archival Reconstitution: Algorithmic Sampling, Visualization, and the Production of Meaning in Large Moving Image Repositories »Signata [En ligne], 12 | 2021, mis en ligne le 31 mai 2021, consulté le 20 juin 2024. URL : ; DOI :

Haut de page


Eef Masson

Eef Masson is a senior researcher at Rathenau Institute (the Hague), a research for policy institute concerned with the societal impact of science and technology. Previously, she was an assistant professor of Media Studies at the University of Amsterdam, where she taught courses in film and media history and media archiving and preservation, and published on non-fiction and non-theatrical films, media archives, museum media, and practices in data-driven research and data visualization. Until the Autumn of 2019, she was a senior researcher in UvA’s The Sensory Moving Image Archive research project.
Email: E.Masson[at]

Christian G. Olesen

Christian Gosvig Olesen is a postdoctoral researcher at Utrecht University and lecturer at the University of Amsterdam’s Media Studies Department, where he teaches courses in film studies, media preservation, restoration and digital heritage. He acted as researcher and project manager in UvA’s The Sensory Moving Image Archive project.
E-mail: c.g.olesen[at]

Haut de page

Droits d’auteur


Le texte seul est utilisable sous licence CC BY 4.0. Les autres éléments (illustrations, fichiers annexes importés) sont « Tous droits réservés », sauf mention contraire.

Haut de page
Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search