Taking Note: Challenges of Dealing with Graphical Content in TEI

This paper describes the ongoing development of a TEI-conformant model for the encoding of hybrid primary sources containing text and graphical components on a similar level of semantic meaning. The subsequent considerations are part of the digital scholarly edition project representing the notebooks of the Austrian conceptual artist Hartmut Skerbisch (1945–2009), a handwritten corpus created over a period of almost 40 years. Besides text, drawings were an important visual means of expression for the artist. In the context of digitally representing the numerous graphics embedded in the notes with equivalent depth to the text, a three-step model is proposed. This model takes into account the (1) graphical components characterizing the composition, (2) textual functions describing and directing the execution of artworks, and (3) interpretation of the graphics and their contextualization with connected material and information. The paper discusses the existing methods of representing graphics in TEI, presents the combination of these methods in the actual scholarly edition project, and introduces the Journal of the Text Encoding Initiative, Issue 12, 29/01/2020 Selected Papers from the 2017 TEI Conference Taking Note: Challenges of Dealing with Graphical Content in TEI 2 semantic enrichment of the TEI sources through formal descriptions in RDF/XML taxonomies using linked open data. Thus, the genealogy of artistic concepts and artworks documented in the notebooks will become traceable.


Introduction 1
The primary focus on text is not only inherent in the name of the Text Encoding Initiative itself, but also in the initial scope of the TEI as an encoding standard focusing on the notion of a text (Ide and Sperberg-McQueen 1995, 7-8;Burnard 2014, chap. 2). Undoubtedly, the TEI has become the means of choice for representing text in digital form in general and digital scholarly editing in particular. Although the TEI has always taken a generalistic approach, considering documents of any size, any number and any content (Burnard 2014), an extensive description of visual materials is nevertheless lacking in the TEI Guidelines. Since, in the wake of the large-scale digitization eorts of recent years, there is an increasing need to deal with hybrid source materials in digital humanities research, the TEI must also meet these challenges. Therefore, extended mechanisms and additions to the TEI Guidelines are required to open and expand the model for encoding nontextual sources with the same level of meaning as text.
This paper discusses a vivid example of such a hybrid source, namely an artist's notebook. In performative, ephemeral, and temporary art forms like conceptual art, performance art, and media art, notes play a key role in interpreting and understanding the actual artwork. They function as a documentary source, one of few remaining witnesses (or, in this case, the only one) of a temporally limited artistic event or artwork. This calls for a comprehensive representation of all related source components including notes and graphics. 4 This article will introduce a TEI-conformant approach to encoding sketches in the notebooks to a similar depth as the text itself. For this purpose, the TEI encoding of the actual record on a descriptive level is semantically enriched by factual statements in taxonomies using linked open data sources. The representation as a graph benets the notebooks' inherent characteristics of distributed information by allowing for the implementation of exible and reusable search mechanisms, the visualization of relationships both within the notebooks and outside, and the indexing of documents in dierent ways.

Project Context, Artist, and Source 5
The reections on hybrid primary sources incorporating text and graphics on a similar semantic level and the challenges of digitally representing these phenomena discussed in this paper are part of an ongoing digital scholarly edition project on the notebooks by the Austrian conceptual and object artist Hartmut Skerbisch (1945Skerbisch ( -2009. It is a digital scholarly edition of the artist's notebooks with a special focus on the genesis of his artistic work. The overall goal of this project is to reconstruct the artist's inspirations powered by external inuences and associative processes in the course of his developing individual manifestations: the evolution of a specic idea and its change over time. 8 In addition to text, the notebooks contain a remarkable number of graphical representations, such as sketches, drawings, or scribbles (on approximately 450 out of 2100 pages). Skerbisch used this form of expression for recording his ideas and for preparing and planning works of art. Thus, they are merely accompanying the text instead of being illustrative, but they carry the same weight and equivalent meaning to the text itself. They can help to trace various stages of the realization of specic artworks and concepts as well to show developments and modications of particular ideas. 9 Figure 1 shows on the left a graphic with a view into a tetrahedron: its top is held up by a rod and an amorphous shape is recognizable on the bottom. It is actually a tent held by a branch, and the human indicated by a lung seeks shelter under the tent. The example on the right shows the same subject, but from an aerial perspective, and supplemented by a circle designated as a stone slab.
The accompanying text gives information on the materials, colors, and objects to be used in the actual installation, and it labels specic components of the graphic.  The complexity increases when dierent views and levels of details coincide on one page. Figure 3 shows a wooden box with an amorphous mass expanding on the bottom as a central representation in full view. Additionally, it shows four detailed renditions of individual areas of the full sketch from dierent angles. The drawings are accompanied by descriptive text and labels. Although the connection between the tent and the box is not given on a formal level, it becomes clear through the investigation of shared intellectual concepts, forms, and materials. These examples demonstrate the need to consider graphical representations as rich information carriers that require a specic description model. The central question is therefore how to develop a "notion of graphics," i.e., how to appropriately treat graphical representations in a text-centric encoding environment.
Journal of the Text Encoding Initiative, Issue 12, 29/01/2020 Selected Papers from the 2017 TEI Conference The transcription and digital representation of the textual entries in the notebooks mainly uses specic transcriptional elements from Chapter 11 "Representation of Primary Sources" of the TEI Guidelines (TEI Consortium 2019) 5 to prepare the documentation of the origin and the genesis of the text. Drawing on the principles of genetic editing, mechanisms have been established by a dedicated working group on Genetic Editions focusing on the physical appearance and the writing process of a document rather than the intellectual structure (Burnard et al. 2010). This led to a thorough investigation of the French editorial school of the critique génétique (Grésillon 1999).
While this movement focuses primarily on the creative writing process of literary manuscripts, Pierre-Marc de Biasi points out that the genetic approach can also be transferred to non-textual material, especially to architecture and sculpture (De Biasi 1997, 124). In the case of the present notebooks used for the conception of artworks, the constitution of the text plays a subordinate role to the unfolding of artistic ideas and general concepts. According to this assertion, the genetic approach is applied to graphical representations. Sketches, formulas, and other graphical representations are integral forms of artistic expressions. Drawings can carry a specic message which cannot always be expressed with words. Here, the graphic becomes the primary object, and the text moves into the background. Representing visual material like graphics is thus not the task of-nor suitably dealt with by-text-image-linking tools, for which a range of satisfying solutions already exist (OpenSeadragon, 6 Mirador,8 etc.). Instead, it is necessary to accurately describe and encode the type and design of graphical representations as well as the subservient functions of text in a formal, comparative, and processable way. How then can the researcher apply the genetic approach to sketches and graphical elements in the TEI context? The sketches in gure 4 exemplify the development between the various graphical representations, which is in fact comparable to textual alterations in genetic criticism.  Figure 4 shows three stages of a tent construction from an axonometric view. The most signicant change can be seen in the type of the supporting structure: the rst construction shows a single rod, placed in the middle, with the textile spread from its top; the second one shows a rod placed in the middle with three rods attached to it, forming a tetrahedron; in the third construction the centre of gravity is shifted and the rod is tilted. These dierent stages have to be formally described.

Representing Graphics with TEI 14
In the following, the current possibilities and their pitfalls for representing graphics in TEI are discussed. Resulting from this consideration, a TEI-conformant solution for a detailed and formal description of graphical representations in a text-centric encoding environment is presented.
Several approaches to encoding photographs, illustrations, graphics, and other visual material that can appear as part of a text are already featured in the TEI Guidelines in chapter 14.4. on "Specic Elements for Graphic Images" (TEI Consortium 2019) 9 and 11.1 on "Digital Facsimiles" (TEI Consortium 2019). 10 The simplest method, providing a mere indication of a graphic's existence embedded in a continuous text, is recorded by the element <graphic> with a reference to the location of the digital surrogate in the @url attribute in form of a URI-reference, as shown in example 1: This simple mechanism is useful in the context of illustrations accompanying the text, where the image has illustrative character and not an explanatory function. This minimal encoding, registering the graphic's existence, can be expanded through nesting the graphic reference <graphic>, the prose description <figDesc>, and the textual transcription in arbitrary text blocks <ab> within a <figure> element employing a structure-oriented encoding approach. Example 2 demonstrates the use of this expansion for the rst drawing of gure 4. When focusing on the physical disposition of the document and the writing (and drawing) process rather than the textual structure, the <sourceDoc> element has to be considered. Example 3 uses the same source and shows the segmentation of textual and visual content into surfaces, zones, and lines with exact coordinates for locating every discrete graphical component and portion of text. <sourceDoc> <surface> <zone ulx="288" uly="136" lrx="1500" lry="892"> <zone ulx="947" uly="193" lrx="1083" lry="257">Stab</zone> <zone ulx="947" uly="193" lrx="1083" lry="257">Tetraeder</zone> <zone ulx="944" uly="325" lrx="1147" lry="378">Zwei Oesen</zone> <zone ulx="942" uly="378" lrx="1206" lry="435">Zwei Leder △</zone> <zone ulx="936" uly="479" lrx="1466" lry="597">Spitze etwa in But neither of these options carries the necessary complexity to model the graphics on the same formal and content-related level as the text in the notebooks. Using these approaches, it is hardly possible to describe their design and meanings, to formalize the complex interactions of text and graphical components, and to record alterations and modications for comparing various draft stages.

18
An attempt to assign meaning to visual material in TEI-encoded artifacts was made by Martin Holmes's Image Markup Tool which provides image annotation within the <body> and corresponds with dedicated zones of the image in the <facsimile> section which in turn refers to descriptive categories stored in the encoding description of the TEI Header (Holmes 2012). Building upon this approach in order to study the genesis of artworks documented in the notebooks, a formalized description in combination with semantic web methods facilitates the comparison of the preliminary drawings and the recording of developments as well as their contextualization with regard to other notebook entries (text and graphics), overall intellectual concepts, and actual artworks.

19
For a comprehensive description of graphical representations, a multi-layered model 11 is proposed, accounting for the various (1) graphical components, (2) textual functions, and (3) editorial interpretations. The rst two levels, describing the graphical components and the textual functions-essentially constituting the material record of the source-are descriptive. The third or interpretational layer allows for commenting and contextualizing the material with links to internal and external entities, and is provided by the editor. This refers to the editorial dichotomy of Hans Zeller in separating the objective facts-the Befund (record)-and its further editing-the Deutung (interpretation)-of the record (Zeller 1971, 50-52). This methodological dierentiation of editorial tasks in philology is close to Erwin Panofsky's theory on the description and interpretation of works of art: his three-level model consists of a pre-iconographic analysis, describing the formal aspects of the work in question; the iconographic analysis deals with the interpretation; and the iconological analysis considers the artwork in its particular cultural and historical context (Panofsky 2006, 37-40).

20
The rst layer of the proposed model describes-independently of a specic technical implementation-the graphical components of a gure. It declares a) the type of the graphic representation (e.g., sketch, constructional drawing, doodle), b) the projection (e.g., front view, plan view), c) the status of execution (e.g., total view, detail view), d) the material of the information carrier (e.g., paper, newspaper, photograph), e) the drawing instrument (e.g., pencil, ink pen), f) the date or time span (to facilitate a chronological order for further investigation on the genesis of the work), and g) the graphical shapes and gures primarily used (e.g., triangle, square, cube, tetrahedron).

21
The second layer records the textual functions. It refers to any explanatory text accompanying the graphic provided by the artist for descriptive purposes. This category includes a) a caption, b) a description related to the whole graphic or parts of it, and c) a label which designates a specic component of the graphic, sometimes made explicit through a connecting line or claried through its distinct positioning. Additionally, the textual content can be of a specic type (e.g., providing information on the material or measurements to be used in the physical manifestation).

22
The third layer represents the editor's interpretation. The model consists of a) a general comment, b) several relations-to other notebook entries, to external entities from literature, art, music, or other preparatory objects (like physical models or photographs), to actual manifestations, and to general theoretical concepts reected on by the artist-and c) the image genesis, which is not explicitly encoded but is a result of the indexing process and therefore operative: variants are generated automatically and can be brought into a sequence to show-in a manner comparable to dierent stages of a text in genetic criticism-the development of modications. The factual information is represented as RDF statements in dedicated taxonomies, enriched with information from existing authority les where applicable.

23
The model in its reduced form neglects those categories that are either already applied as general metadata to the notebook (e.g., author, archive, copyright information), or those that do not apply to Skerbisch's graphics. If required, the model can easily be extended.

24
The application of these categories to the tent in gure 4 is shown in tables 1, 2, and 3.
Constants and alterations are easily identied and documented. Most of the properties-like type, projection, shapes, material, notebook entry, external reference, manifestation, and conceptrefer to thesauri and controlled vocabularies from existing authority les, especially the Art & Architecture Thesaurus (AAT) 12 and the individually compiled catalogue of Skerbisch's artworks 13 and artistic concepts. 14 Tables 1, 2, and 3 reveal that many of the instances remain the same across the three representations, but some show distinct dierences, especially when it comes to the textual functions which contain instructions for materiality and measurements for the physical realization of the artwork.  To transfer the demonstrated model into digital structures, the existing methods for encoding graphics in TEI-already introduced in the paragraph on Representing Graphics with TEI-are brought together and enhanced with a semantic web approach. The proposed means of encoding the graphic from gure 5 in TEI is as follows: <!--section 1: encoding of the physical dispositions in the facsimile structure --> <facsimile> <surface xml:id="fol_19v"> <zone xml:id="F-27" ulx="316" uly="145" lrx="1482" lry="837"> <zone xml:id="F-27-01-a" ulx="947" uly="193" lrx="1083" lry="257"/> <zone xml:id="F-27-01-b" points="595,515 610,194 620,193 615,514"/> <zone xml:id="F-27-02-a" ulx="947" uly="267" lrx="1143" lry="317"/> In the TEI Guidelines, <label> is dened as containing "any label or heading used to identify part of a text, typically but not exclusively in a list or glossary" (TEI Consortium 2019, chap.

<label>).
18 This supplementary phrase allows for a broader interpretation and the use of the <label> element in this context. In case the artist gave additional instructions concerning measurements and materials for the planned installation, they are encoded using the elements <measure> and <material>, although the latter was originally introduced for describing the physical material of manuscripts (TEI Consortium 2019, chap. <material>). 19 The relation between the intellectual content and the physical disposition is established by the @facs attributes on the gure tagset, pointing to the designated @xml:id attributes on the corresponding <zone> elements. This largely covers the description of the text functions, but the formalization of the graphic-specic properties is still missing. Descriptive elements from the TEI/XML encoding are semantically enriched and formalized by linking them to dedicated statements on subjects in taxonomies expressed in RDF/XML using linked open data sources. This is implemented using the @ana attribute which "indicates one or more elements containing interpretations of the element on which the @ana attribute appears" (TEI Consortium 2019). 20 This trinity of image, sign, and meaning of content (Vogeler 2015) facilitates a) the formalized representation of the TEI-encoded content, b) the retrieval of common factual statements, and c) the visualization of relations between dierent entities. The next step will be to formalize a special case of the <figure> element in a project-specic ODD in order to further implement the proposed model. In example 4, the value "art:A10003" refers, indicated through its prex "art", to an RDF triple of a description from the artwork thesaurus, which contains all known artworks by Skerbisch, with related information.
The prex "gt" addresses subjects from the graphic thesaurus, which incorporate concepts from existing authority les in addition to individually dened concepts where the authority les are lacking. It can be exibly expanded at any time when new material is edited. The graphic thesaurus represented in SKOS currently contains seven top concepts: 1) visual works, 2) materials, 3) drafting, drawing and writing equipment, 4) supporting material, 5) geometric gures, 6) views, and 7) interpretation. In summary, this approach of a) localizing the topographic aspects of the graphic in the <facsimile> section, b) graphic description through <figure> elements embedded in the text structure, and c) semantic enrichment using the @ana attribute allows for the detection and extraction of graphics related by their content throughout the entire notebook corpus.
Comparing the changes with a formal model clearly shows the alteration of key components and concepts, i.e., image genesis. Hence, the encoding of related concepts allows the extraction, comparison, and genealogical sequencing of related sketches from throughout the corpus.

26
Similar concepts assigned to graphics can be retrieved through SPARQL queries: during the upload process into the repository, information encoded in the TEI document is formalized as RDF triples which are stored in a triple store for further retrieval.

27
Example 5 shows the RDF triple with the actual statements representing a single graphical representation (see gure 5) from the notebook, indicated as a URI in the @rdf:about of the the position in the notebooks gives relative temporal information to create a chronology of changes. This is implemented as a search function, dynamically transforming the results to a web representation. Drawing on the encoding of the location of graphical details explained above, single components are highlighted using JavaScript and thus support the interpretation process (gure 6). This perspective on the corpus draws a picture of the chronological sequence in which the graphics occurred and shows the intensity with which the artist has devoted himself to a specic topic in the conception of his artworks. Since the notebook entries are also linked to artworks and artistic concepts, this method allows searches for corresponding text passages in addition to the graphics.
Through the combination of textual entries, graphical representations, external links, and work manifestations, a comprehensive analysis of the work creation process becomes possible.

Closing Remarks and Future Developments 30
A quotation from Hartmut Skerbisch himself aptly summarizes the challenge at hand: "Though not self-evident, there are after all also works of art. Once we get involved, we are in a labyrinth of relationships and seek a way out" (Skerbisch 2000). 31 Skerbisch's artworks usually evolved over a long period of time. This becomes evident in the notebooks themselves: various text fragments and graphical representations are scattered throughout the corpus, and ideas undergo numerous variations. Based on the dierent manifestations shown in gure 7, it is a challenge to understand the ideas behind them without familiarity with the artist. The tent appears in a private photograph from 1969 and is taken up again in the artworks Erde. Our cubehouse still rocks from 1976 and yet again two years later in the installation Lispn!. The latter refers to a quotation from James Joyce's Finnegans Wake "Soft morning city! Lsp! I am leafy speang Lpf! … Lispn!" (Joyce 1947, 619) which also reiterates reden blattartig, a 1976 literal translation of leafy speang into German. All of these artworks share the concepts of "breathing" and "body" (incorporating the shape of a human lung) and "settlement" (expressed through the tent).

32
Investigating the graphical representations in notebooks-and especially their changes and alterations in material, positions, components, proportions, etc.-can assist us in solving the bigger puzzle of reconstructing an artist's creative processes. Such investigations should therefore be much more prominent in the genre of digital scholarly editing. To achieve that goal, however, it is essential to encode graphics with a similar depth and complexity to text. For this purpose, a combination of TEI encoding and semantic web technologies seems to be a promising solution.  rocks, 1976;Lispn!, 1978. (Photos: Hartmut Skerbisch). 33 The proposed encoding model for graphical representations is currently a prototype, which is applied to the whole notebook corpus. Additionally, its suitability to comparable source material which shares similar characteristics is yet to be investigated. One can imagine that notebooks by scientists (e.g., laboratory books by Alexander Fleming), 22 artists (e.g., sketches and notes by Albrecht Dürer), 23 scientic travel correspondence (e.g., Alexander von Humboldt), 24 notes from architects, engineers, and even medieval manuscripts will benet from the application of the proposed model. Only through the combination of an equally thorough encoding of textual and graphical components, and their semantic enrichment, can such hybrid source material be fully understood.