1If we keep zooming in on a digital image, before long, we will reach a pixel. A pixel is a block of color of nothing, a set of RGB values, and the resolution limit of a digital optical device. Here is where human vision ends, but where the story of seeing something where there is nothing begins. On the planetary scale, the resolution limit of satellite sensors conceals objects as large as trees underneath a pixel. Remote sensing scientists switch between ground-level observation and aerial view to mathematically decipher that which the spatial resolution limit of satellites has rendered invisible.
2The video essay contemplates the collaboration between me, a photographer, and a remote sensing researcher, Daniel Schraik, during a field trip to the Hyytiälä field station. Following the research group who had set out to overcome the spatial resolution limit of satellite imagery, the essay elaborates on the way scientists utilize ground truth data and on-site observations to validate the data collected by satellite.
3In everyday language, the term ground truth refers to a first-hand experience. Ground Truth connotes the documentary tradition and the act of witnessing. In remote sensing, however, ground truth relates to data collected on-site, which are then used to calibrate, to build models, to predict, to interpret, and to decipher information from images—in this case, satellite images. Similarly, the collaboration reveals another operational layer of photography beyond the immediately visible, illustrating a trans-scaler vision composed of mathematical models, sensorial data, photographic evidence, human observers, and computational agents. The combination of real-life video footage and the point cloud of a forest from the scientists’ dataset contrasts the representational approach of drawing and photography with the data-oriented and algorithmic approach of computer-aided seeing. The parallel reading of the same landscape contextualizes an epistemological regime that emphasizes computational models rather than optical lenses and decenters humans as the primary observing agent.
Figure 1. Sheung Yiu, Spotting a Tree – shots from the film.
© Sheung Yiu.
4More than two decades after Harun Farocki’s Eye/Machine trilogy (2001-2003), the seminal video essay exposing the weaponization of images in modern warfare, much has been said about the paradigm shift in photography. In the essay, images are the raw material of military intelligence analyzed under techniques such as mapping and object recognition. The video essays reveal the way images are becoming part of the production process – and that includes computation – rather than the products of the process. Farocki distinguishes the emerging genre of images, which he calls “operational images”, from images meant to represent, entertain, and inform (Farocki, War at a Distance). The notion of operational images opens new avenues for photography theory to think beyond the dialectics of truth and representation. Rather, in the algorithmic turn of photography, technologies and the surrounding operational protocols are worth a more in-depth study in and of themselves (Uricchio 11). One of these avenues that Farocki has provoked is an interest in the hyperconnected system, or indeed an ecology, in which images reside. Images are embedded in a much larger system and a confluence of interests in which a visual analysis provides insufficient explanatory power. Media theorists Adrian MacKenzie and Anna Munster have put forward the concept of “platform seeing” as a new mode of invisual perception, enacted not by an observing “subject” but “observation events distributed throughout and across devices, hardware, human agents and artificial networked architectures” (MacKenzie and Munster 3). On the other hand, Daniel Rubenstein and Katrina Sluis's analysis of the networked image highlights the role of metadata as an “invisual” protocol in organizing images (Rubenstein and Sluis 18). Both theoretical approaches point towards a post-photography era, in which the entanglement of photography, data, and invisual processes are the norm and not the exception.
5Thinking with and through these ideas, my video essay Spotting A Tree From A Pixel unpacks these operational processes and how images are involved in these processes. I focus on processes in disciplines that lie at the intersection of art and science computation and photography, namely remote sensing, 3D computer graphics, and computer vision. The video essay is a reflection of my three-year collaboration with scientist Daniel Schraik who works in remote sensing, “the science of near real-time environmental monitoring using open access data without on-site observation” (Schraik 1). Remote sensing evolves from a long history of forest surveying techniques, which combine aerial photography (and, more recently, satellite imagery) with field observation to assess the ecological features of forests. Fieldwork complements imagery intelligence by verifying details not captured on the images, thus creating a conversion table between features observed on the ground with data captured on the image. With the conversion table and knowledge about optics and perspective, foresters can extract useful information about a forest from its image alone. For example, a 1-cm shadow on an image created by a pine tree at noon captured from 3 kilometers above ground translates to a tree height of 30 meters.1 The conversion table makes forest management scalable2; i.e. information on a grid can be applied and extrapolated to all other grids of the same image, and the same extrapolating methods can be applied to all other images taken in similar situations with the same patterns.
6Schraik and his team push the practice one step further by leveraging advanced computation and imaging techniques. They propose to improve the interpretation of satellite imagery by developing a better forest reflectance model. A forest reflectance model mathematically encodes the relationship between sunlight and forest. The relationship is established by meticulously measuring, for example, the effect tree structure has on light transmission. His research focuses on developing a model to simulate and infer details about the forest without the need for on-site observation. Ironically, this required a large amount of ground truth data and field trip observations to verify results. One experiment involves 3D scanning a tree seedling, then cutting it down into pieces, weighing and scanning each leaf and branch to calculate the correlation between tree structure and photosynthetic surface area. To reiterate in less technical terms, if the relationship between sunlight and forest – i.e., streams of photons from the sun enter, scatter, reflect back from the forest, and are finally captured by the sensors in the satellites – is known, then scientists can extract much more information about the forest just by satellite data alone, even that not immediately visible on the image. Metaphorically speaking, one can resurrect a tree from a pixel of data. His research on forest reflectance model thus illustrates how the entanglement of computation and photography offers new ways of perception and knowledge production.
- 3 The use of the term “afford” is a reference to J.J. Gibson’s concept of affordances, which connote (...)
7In between mathematical models and meticulous field measurements, the concept of scale underlines the discipline of remote sensing. Firstly, real-time planetary observation is limited by the scale of observation that satellites can afford3. On the surface, it might seem that aerial photography has captured everything at which the lenses are pointing when, in fact, details are finite on the image. Every digital optical device has a resolution limit; a point at which two sources of information can no longer be resolved. On a digital image, the resolution limit is the pixel, a block of color defined by a set of RGB values. The resolution limit of an optical device depends on its technical specifications as well as the wavelength of light with which the observation is made. Most wavelengths useful for planetary observation have a spatial resolution of at least ten meters. In other words, each pixel of the image represents a ten-square-meter plot of land on Earth. The resolution is sufficient for remote sensing scientists to determine the land type but not enough to differentiate individual trees. A tree is hidden underneath a pixel. Consequently, scientists combine field observation and satellite imagery, triangulating information from both observations to obtain a fuller picture of the forest ecology.
8Secondly, the researchers circumvent resolution limit by developing a model – a forest reflectance model – to translate observation from one scale to another. A forest reflectance model is a set of mathematical equations describing the interactions between sunlight and the forest. By meticulously measuring, scanning, weighing, and digitizing the forest, scientists establish a statistical correlation between the tree and the signal it produces on the satellite sensors. These measurements are conducted on multiple scales, from a leaf to a branch, a branch to a tree and a tree to a forest. With each measurement, the scientists revise the models, adding new variables and adjusting parameters to better mathematically formulate the optical essence of trees. To put it in terms of scale, researchers are capturing observation in one scale as a model and applying in on another scale – trans-scalar seeing. As philosopher Joshua DiCaglio put it in his book Scale Theory, “At the scale of the planet, as forests give way to colors on a landscape, trees are no longer observable in any way that could clearly be tied to the current observation except via a scalar shift.” (DiCaglio 56) At the same time, the model makes forest imagery intelligence scalable to other forests. Observation on a smaller scale can be applied to a larger-scale landscape. In remote sensing, scale is both an obstacle to overcome and a scientific technique of seeing.
9Media theorist Zachary Horton wrote an entire book to unpack the notion of scale. In Cosmic Zoom: Scale Knowledge, and Mediation, Horton differentiated scale from size and pointed to the ontological connotation of scale.
The nanoscale can be as small as the head of a pin or as large as a galaxy; its spatial extension is arbitrary. What marks it as the nanoscale is the typical or characteristic size domain of its entities and dynamics. The nanoscale is the domain in which features measured in nanometers can be resolved as individual entities […] (Horton 16)
10To continue the discussion of scale in remote sensing with this in mind, a satellite image captures a huge strip of forest, but the resolution of this general picture is not in the scale of individual trees. The satellite creates a digital image, a grid-like map on which each 10mx10m plot of land is represented as a discrete set of RGB values. The individual entities in this mode of planetary observation are ten-square-meter plots of land. The grid marks the limit of the amount of detail that the image can reproduce. In Horton’s word, “scale expresses this tradeoff between size and resolving power.” (Horton 15)
11In my artistic research, I argue that scale is crucial in understanding contemporary visual culture underlined by the explosion of machine-generated imagery and advanced computational infrastructure. Remote sensing is emblematic of this scalar milieu and the paradigm shift in image culture in general. In hindsight, the connection between remote sensing and contemporary visual culture is not coincidental given its historical entanglement with forestry and its assemblages of technique from data gathering to statistical inference. For example, phytogeography, a technique in German forestry of partitioning of space into same-sized grids for evaluating forest ecology, shared conceptual proximity with modern computation practices (Parikka and Gil-Fournier, 2021). The technique of abstracting landscape is also comparable to how contemporary digital airborne photography and satellite imaging abstract a landscape into a map of pixel values. Furthermore, the data abstraction laid the foundation for sampling and statistical inference in environmental science, the techniques which are now being applied to a broader type of visual imagery in the field of computer vision, computer graphics, and cultural analytics. Remote sensing stands in the confluence of technological development, statistical innovations, and image interpretation.
12In photography, the question of scale manifests in image resolution, capturing, resolving, and interpreting details from an image. Web 2.0 social media, portable smartphone cameras, and omnipresent automated imaging devices have drastically increased image production. The ocean of images online becomes the source of data for analysis, but the sheer amount of data (i.e., size) does not guarantee a better understanding without an improved understanding of scale (i.e., resolving power). Big data is big but remains incomprehensible until it is contained in the appropriate scalar domain. The explosion of image production and their hyperconnectedness made “visual data” readily operational by a myriad of innovative statistical techniques and applications. Computer vision leverages the power of machine learning for pattern recognition, while remote sensing circumvents the limit of information set by resolution through statistical modeling. Similarly, visual culture is experiencing a statistical turn, and scale is crucial in understanding the breakthrough in image interpretation: how scale creates new ways of knowing.
- 4 Hito Steyerl, “A Sea of Data: Apophenia and Pattern (Mis-)Recognition,” accessed April 12, 2021, h (...)
13The video essay reveals the statistical mode of perception with equal fascination and skepticism. As a means to transcend scale, statistical modeling combined with the digitization of images provides novel ways of sense-making. Remote sensing applies the forest reflectance model to image interpretation, but the underlying computational logic is present in predictive models across disciplines from climate science to epidemiology. Computational models extract even more information than the resolution limit can afford, affording scientists the power to access temporal and spatial scales beyond the immediately visible and humanly comprehensible. As human perception is being abstracted by computational analysis as the standard of knowledge-making, uncertainty brews between the trade-off of scalability and comprehensibility. Among all these is the fear of algorithmic apophenia, the perception of patterns within random data (Steyerl).4 Apophenia is the uncertainty of the information we are looking at, whether it is a meaningful pattern or just random noise. As I questioned at the end of the video essay, “Our ability to collect and discern data promises the power to resurrect a tree from a pixel. At that moment, do I see a tree? Or, am I seeing something where there is nothing?”
14
RSS feed
