Many thanks to the anonymous reviewers, who helped to improve this text considerably.
1At the recent “Encoding Cultures—Joint MEC TEI Conference 2023,” Roeder et al. (2023) showed the difficulties in defining an appropriate encoding and rendering for historical documents in music theory, here: major publications from the dualist/monist controversy in the nineteenth century. Challenges lie in the large (sequences of paragraphs, music examples, graphics, tables, etc. integrated in the text flow) and in the small (references to pitches, pitch classes, keys, graphics, etc. interspersed with running text).
2Roeder et al. demonstrate the challenge of encoding historical documents of this genre, for instance with the long forgotten but still relevant book Die Natur der Harmonik und Metrik by Hauptmann (1873): Single-character abbreviations stand for pitch classes, while the case of the character encodes the syntonic comma; the horizontal hyphen and the vertical bar are not just typographic symbols but well-defined operators between pitch classes, etc. Even more challenging is supporting contemporary authors, who often want (a) to use a self-defined, much more flexible and extensible set of entity symbols, cross-reference systems, expression grammars, etc., and (b) to write their text in a creative flow, in which file identifiers, correct XML tagging, editor dialogs popping up, etc. are hardly bearable distractions. Furthermore, (c) presenting analysis results and corpus interpretations as dynamic documents in an interactive way is an advanced contemporary means to demonstrate complicated interdependencies to the reader more clearly: see for instance the animated and transmorphing views of sketches by Beethoven generated recently by Kepper et al. (2023).
3The text format definition and processing pipeline presented here is called multiNotes. It has been developed for one particular medium-scale analytical project and addresses the goals (a) to (c) listed above. The code can be reused as it is for further projects.
4multiNotes is based on d2d and standard processing tools: gnu make (FSF contributors, 2023), bash (FSF contributors, 2022), LilyPond (Nienhuys and Nieuwenhuizen, 2003; LilyPond, 2011), PostScript (Adobe Systems, 1999), ECMAScript (Guo et al., 2023), and netpbm for simple graphic transformations (Henderson,2023). D2d is a text format definition and compiler-based implementation to allow domain experts and novelists to write texts in the flow of authoring, with minimal technical interference, that are nevertheless valid XML documents (Lepper et al., 2001; Lepper and Trancón y Widemann, 2011, 2016 2019). Since both a d2d text itself and its type definition are just another encoding of XML, the backends of the rendering pipeline can be changed in future from direct XHTML plus LilyPond plus PostScript to something more abstract like TEI plus MEI plus SVG. The architectural design and the challenges encountered would still apply as described in what follows.
5The architecture described here helps authors create interactive documents that interleave fragments of music notation (in traditional CWN, plus possibly analytical graphics) with paragraphs of commentary in running text. It is called multiNotes because it allows multiple views of multiple segments of music notation by interactively hiding all or parts of the commentary, the music notation, and the analytical graphics.
- 1 Throughout this article, “lines of code” (loc) are measured by applying the tool cloc(...)
- 2 This study applies in nearly all aspects technologies that are complementary to those (...)
6It has been developed for one particular use case, namely a medium-scale analysis of the first movement of the Third Symphony by Gustav Mahler (Lepper, 2015) (120 pages approx. when printing only the text layer; 18,927 lines1 of source text).2 The mandate for that analysis was given by Mahnkopf (1996, 37) who stated that “Mahler, der Meister der unähnlichen Ähnlichkeit, fordert unter den Analytikern den Phänomenologen, nicht den formalästhetischen Akademiker.” / “Mahler, the master of unsimilar similarity, requires the phenomenological analyst, not the academic and formal aesthetician.”
- 3 Of course this concept is philosophically disputable, perhaps even impossible and obs (...)
7The author’s idea was to write a purely phenomenological text that would closely follow the reception by a virtual “ideal listener without any contextual knowledge.” This “purely phenomenological” text thus follows the execution time of the music in a linear fashion, and tries to describe the reactions of that “ideal listener” who knows neither future of that piece nor anything about related compositions.3
8A full-fledged analysis, however, must contain contextual information. Only this kind of text knows about the historical circumstances of the composition and the upcoming measures of the running piece. Therefore the text has been split by markup into a “purely phenomenological” and a “context-aware” layer. This normally separates complete paragraphs or sentences, but can also cut into a sentence, as in “Der folgende Formteil #ZUK wirkt als Rf zur Rp und #/ZUK bringt grundsätzlich eine zweistimmige Struktur.” / “The next section [functions as transition to the recapitulation section and] brings basically a two-part structure.”
Figure 1. Screenshot of the different components as rendered by Firefox.
Figure 2. Simplified grammar view of a multiNotes document. (Specific elements injected into the generic article definition appear in boldface—see table 1.).
9Figure 1 shows the different kinds of components of a multiNotes document, as rendered by a Firefox browser. The nonphenomenological text components are printed on a gray background. Interspersed with this commentary text, the document consists of an excerpt of the work under discussion. In the Mahler analysis, music notation and text layers are of equal weight, with respect to the source text as well as the visible result.
10This music notation layer offers the reader an additional feature: by switching off all textual components completely, it can be used as a listening companion. By adhering to strict rules, this excerpt indeed contains every single sounding pitch of the full score, while abstracting from dynamics, articulation, instrumentation, and partly from duration (but not from octavation). It is split into multiple segments which appear interleaved with the paragraphs of the analytical text. The excerpt’s rules are listed completely and in detail in the publication; see its section 1.4 (in German).
11For example: voice lines set in multiple-octave registers are only shown once; all other octaves are indicated by adding to the very first note a diamond or cross-shaped note head; arpeggiated accompaniments are shown as held chords, with tie arcs omitted. All these rules aim at an easily readable but complete representation of all harmonic, diastematic, and contrapuntal events. The analytical text refers to the excerpt segments, and vice versa the note segments contain analytical graphical components (brackets, motif identifiers, accent marks, arrows, etc.)
12The multiNotes source text is given as one single d2d document which adheres to the corresponding multiNotes format definition. A link to this source is at the very end of its HTML rendering. A navigable documentation for the multiNotes format is generated by the d2d system and found online in English and German. The format and markup definition is derived from a standard general-purpose text model, implemented as a d2d basic library module, and the definitions in multiNotes.ddf are derived by rewriting, as can be seen by the import statements at the beginning of the document type definition file. Therefore it inherits the full markup for conventional texts: header and metadata entries; physical markup of running text like emphasized, italic, bold; general semantic markup like persons and opera; simple tables and lists; embedded hyperlinks, text-internal labels and jumps; top-level structure with parts, chapters, sections, and appendices; floating objects and their List of Figures (LoF); inclusion of images and sources for arbitrary processors like SVG, LilyPond or LaTeX; footnotes; and bibliography. Many of these general features are not needed or not specific to musical analysis. Therefore figure 2 gives a very simplified grammar, containing only the elements of particular interest. Nevertheless, the floating objects, tables, and embedded LilyPond sources (as inherited from the general-purpose format) are used beside the specialized elements; see the LoFs at the beginning of that study.
13The very last link in the web page points to its source text.
- 4 One single output file embed.sh is generated, which contains “here-documents” split by bash, only t (...)
14As mentioned above, the implementation of multiNotes uses a collection of tools from the public domain. Figure 3 shows the processing pipeline: All input files are shown with a bold border. The input and output for one particular analysis are filled with gray; all others constitute the reusable infrastructure. The d2d source text is translated into one XML text file. This is processed (under the control of gnu make) by XSLT into one XHMTL main file, which is the medium finally presented to the reader by an arbitrary browser, and multiple source files to be translated by LilyPond into graphic files included by that main file.4 LilyPond also generates the analytical graphic and text addenda by its integrated PostScript escape mechanism.
15The complete infrastructure of multiNotes is realized by:
-
Makefile to coordinate all processing steps (132 lines of code).
-
multiNotes.ddf in d2d’s document definition format (ddf) defines the grammar of the input text and all XSLT rules for its processing (495 loc).
-
multiNotes.ly, included by all generated LilyPond sources, contains LilyPond layout directives and macro definitions, for note setting and the additional PostScript graphics (97 loc).
-
multiNotes.js , included by the generated XHTML file, realizes the API for the dynamic behavior (104 loc).
-
textnav.js , included by the generated XHTML file, realizes the API for the navigation chains (230 loc).
-
multiNotes.css is included by the generated XHTML file (21 loc).
16All source texts are in the public domain; a publication on an open-source code repository is in preparation.
Figure 3. Source files and processing pipeline of multiNotes .
17Interactively controlling the visibility of textual and musical elements is crucial to creating web outputs that flexibly support variant readings. This section outlines how this is implemented in a multiNotes document. The main .html file generated by multiNotes is interactive and dynamic, employing the programming features of ECMAScript operating on the document object model (DOM) of the generated XHTML (Guo et al., 2023). The ECMAScript code provides (a) multiple kinds of internal links which support sophisticated navigation (see section 5), and (b) that the visibility of text layers can be switched on and off. When starting the presentation, all components are visible. They can be switched on and off by dedicated active links , the targets of which are defined to call the API provided by the ECMAScript file. The nonphenomenological, context-aware parts of the text are shown on a gray background (see figure 1). They can be switched off by one of the active links, thus showing the purely analytical text as demanded by Mahnkopf (1996), or all analytical paragraphs can switched off completely, showing only the notated music.
18Most excerpts from the exposition show the corresponding version from the recapitulation section in additional staves; all excerpts contain additional analytical graphics or text. The output can include a check box to show or hide each of these components, and there are global active links to set the visibility of all of them. Figure 4 shows a screen shot of an exposition excerpt with recapitulation correspondence and analytical additives displayed.
Figure 4. Fragments of source text and screen shot of an exposition excerpt with recapitulation correspondence and analytical additives.
- 5 Please note that these elements are fully compositional, which allows the lean notati (...)
19To make the note symbols’s visibility switchable at runtime in ECMAScript, their categories must be defined initially in the source text. This supports the grammar of the mNotes element, see figure 2, lines 23–25: The elements E, R, and W stand for the note text in the exposition, the recapitulation section, or both (= “shown alWays”); the lower-case variants mark the corresponding analytical graphics.5 When displaying each score segment in a browser, a nonempty combination of the flags E and R, plus an optional flag A, controls the current display state individually. Material in the element W is always shown; E and R appear only if flag E or R is set, respectively. The same holds for the lower-case variants, requiring flag A to be set additionally.
20In the first third of the text, which deals with the exposition (according to the linear reception of the “ideal listener,” as described above), the excerpts are defined by writing these mNotes elements. In most cases they contain the LilyPond source for both exposition and recapitulation. When rendering, the Flag E is fixed to true, and R is controlled by a checkbox, if such a layer exists.
21In the last third, dealing with the recapitulation, this note material is reused by the elements mNotesRef (see figure 2, line 26). It identifies the above-standing source text by its first measure number and renders only its recapitulation layer: flag E is fixed to false and R to true.
- 6 Because of the special construction of the Mahler III/1, the note examples in the beg (...)
22For new material in the recapitulation that has not appeared in an exposition excerpt, simply a new mNotes element is defined, of course without any #R layer. The same holds for all note text in the development section: the visible layer is always called #E, despite the real meaning. 6
23In any case, flag A is realized by a checkbox if such material exists. The checkboxes can be set and cleared altogether by the above-mentioned active links.
- 7 This makes it complicated to save a complete copy of such a work to a “wayback machine,” etc.
24The XSLT code extracts all required combinations of layers into different LilyPond source files, the names of which reflect the combination of the flags as character codes (_E or _ER or _ERA, etc.) The change in any such checkbox calls the ECMAScript API, which calculates the required file name and loads its graphical content.7
25Figure 4 shows a typical fragment of source text and its rendered result: The few framing statements in blue are d2d source. The majority (in black) is LilyPond source. The red d2d tags limit the layers. Shown in green are interspersed calls to the macros defined in multiNotes.ly that print green motif brackets over the notes (lines 16, 18, 19, 22). Line 27 shows directly entered PostScript for a graphic additive (not in the screenshot fragment), switched on by the red $e.
26Line 32 shows that when an additional half-measure appears in the synchronously running recapitulation text, a corresponding space must be inserted in the exposition text as soon as flag R is set. Line 22 is special, as the analytical switch $e inserts a motif bracket, which is replaced by another with different geometry (y-dist -24 instead of -16) whenever the recapitulation text is visible. This shows how negation can be realized: the $R switch enables the LilyPond-level comment lead-in %%, which suppresses the second motif bracket in that line.
- 8 Here d2d’s default escape character # is changed to $, because the former is required in the $term! (...)
27More complex analytical graphics can be realized by programming language stacking: in the extreme case, d2d is converted to XML and to XSLT, which extracts LilyPond source files in which embedded scheme code produces fragments of PostScript. See figure 5 for an example of a deeply stacked case.8
Figure 5. Language stacking for generating graphic addenda programmatically.
28While the preceding section covered the integration “in the large,” this section speaks “in the small.”
29In any domain-specific text architecture there are semantic entities, the names (or other textual representations) of which are interspersed in the flow of text. Entities of the same or similar type can form “chains” along which readers might meaningfully navigate the document. Entities fall into two categories: Anchored entities are related to some central explanatory text, to which one might want to navigate from any other reference to this entity. Citations are a typical example: they are anchored at the corresponding entries in the bibliography. Another example is indexed notions that also have explanatory entries in some glossary. Navigating to such anchors is generally well understood and familiar to users. Nevertheless, it may be appropriate to navigate additionally along the chain of references. With unanchored entities such a central point is not given, but navigation may nevertheless be sensible following this chain. Table 1 shows the entities currently supported; for their input grammar see again figure 2.
- 9 The anchored entities “motif” and “index entry” are implemented as HTML <a> elements (...)
30In detail, the HTML view generated by multiNotes behaves as follows: Whenever a navigable entity reference is clicked, a navigation window pops up in situ (i. e., near to its current position on the screen). It has clickable areas to go to the first, the preceding, the following, or the last appearance of this entity in the text. The first two are not present at the very first appearance, the last two at the very last. “Go to first” is not present at the second appearance, since “go to previous” has the same effect, and similarly for “go to last.” Figure 6 shows a screenshot.9
Table 1. Domain entities to use inline in sentences.
mot, motDef |
usage and definition of motif abbreviations |
T, TT, TTn |
single metric positions (“Takt”), or ranges thereof |
refN |
reference to a music excerpt |
refRp |
reference to the recapitulation part in a music excerpt |
ta |
a tonal key |
tiefton, hochton |
maximum and minimum pitch classes |
index |
general-purpose indexed word |
nbrev, n1, n2, n4 n8, n16, n32, n64 |
duration note symbols: brevis, whole note, 1/2, 1/4, 1/8 etc. |
n88 |
two 1/8 note symbols connected by a beam |
p4, p8, p16, p32, p64 |
duration pause symbols: 1/4, 1/8 etc. |
nothird |
symbol for “no minor nor major third present” = □ |
arrn, arrs, arrne, arrse |
arrow symbols: north, south, northeast, southeast |
volute |
old-fashioned separator of paragraph sequences |
ZUK |
nonphenomenological but context-aware text section, knows the future = German “ZUKunft.” |
Figure 6. Screenshot of the navigation menu, popped up for measure search.
- 10 If the explanation in the glossary entry for Cxxx refers to entry Bxxx, then naturally this is not (...)
31Clicking in the middle of the five areas makes the menu disappear, with the intuition “I have arrived where I wanted to go.” Due to the overall text organization, nearly always the very last position in each navigation chain coincides with the anchor, if present.10 These chains are implemented for index entries (with glossary entry as anchor), tonal keys (“Tonarten,” no anchor), “Hochton” and “Tiefton” (= local maximal and minimal pitches, no anchor), measure numbers (no anchor), and motifs (anchored). (Citations are anchored in the bibliography, but have no navigation chain.)
32Whenever the text selected through the navigation window is currently not visible due to the text selection state (see preceding section), the menu turns red and no scrolling happens, but the menu is operative as if the jump had occurred.
33Some navigation chains are treated specially:
34Measure numbers: Every such reference goes either to the number of a single measure, or to a range. The tags and elements for this are T, TT, and TTn (see table 1). The last form is needed for the dative case in German: siehe #T 4-7 delivers “siehe T.4–7” (“see ms. 4–7”), siehe die #TT 4-7 delivers “siehe die Takte 4 bis 7” (“see measures 4 to 7”), whereas vergleiche mit #TTn 4-7 delivers “vergleiche mit Takten 4 bis 7” (“compare to measures 4–7”).
35Finding the left and right neighbors for navigation is special: The direct predecessor or successor is the next measure number reference that has a nonempty intersection with the start position of the current navigation journey. So if we click one of the links above, we will get as possible jump targets all references that contain at least one of the measure numbers 4, 5, 6, or 7, like “6” or “2 to 4” or “1 to 10.” This set-based navigation makes sense because when the reader shows interest in one or more of the measures 4 to 7, the software does not know which of them in particular, so any mention of at least one of these is potentially of interest. The original starting point of the navigation journey is always printed in the center of the navigation menu, because it often differs from the text at the current position, which could be confusing: When we land at “1–10” while the original start point is still “4–7,” a neighboring reference to “1–3” or “10” will not be selected. But a click at the center of the menu makes it disappear, and a second click nearby on the measure range starts a new navigation looking for “1–10.”
- 11 The post-dot part can be measured in quarter notes or in half notes, may start counti (...)
36D2d uses its character-parsing power to extract the measure numbers from the source text; see the definitions in figure 2, lines 28–29. The dotted form #TT 7.2 stands for “measure seven, second beat.” It is rendered verbatim as Takt 7.2, and its meaning is not defined formally but left to the context. The formal analysis and navigation logic of measure numbers only deals with the pre-dot part.11
37Motifs: In our Mahler analysis, motifs and their identifiers play a central role. They are treated as anchored entities: At the end of the text is a glossary of them, combining their abbreviations and their descriptions, and also starting a navigation chain. Each motif identifier has a base part and additionally a variant part and a trailing number, prefixed by / or . (see figure 2, lines 33 to 36). The first separator is not printed; the latter is. The navigation chain is based only on the base part, so that “J'.2,” “J'/a,” and “J'/**.3” will all be linked to the motif table entry »J'« and be part of the same navigation chain. The set of allowed decorations in the base part is limited, because they must be replaced explicitly to match the “ID”-restrictions of XML. The numeric suffix is meant for numbering the occurrences in diachronic order, but this is not formalized.
- 12 In the current implementation, the user is responsible for avoiding duplicate IDs tha (...)
- 13 If it appears in more than one.
38The motDef allows an additional explanatory prefix to be printed as part of the reference text, as in main motif!J/1—the motif register will link back to this text position as Erstauftreten Analyse (= first appearance in analytic text).12 The accepted input text for all variants ends with the first blank character, which is discarded, or any other character not allowed according to their syntax definition, which is left to parsing. At the end of an article, the rules for motivGlossary and motivGlossaryEntry set the anchor texts for the motifs: code gives the motif base code, exc the number of the excerpt where the motif appears the first time,13 and shortdesc a concise description of one line. This will also be used for the tool tips of all references in the navigation chain.
39General index entries have a slightly simpler structure; see line 40 in the table: If the subelement text is present, it is used for printing and code for the navigation logic. Otherwise the latter is used for both. Both definitions only differ in the rendering: index is for all in-text references; indexEntry starts the final glossary text.
40The multiNotes architecture was developed for one particular music analysis project (Lepper, 2015). Nevertheless all code (d2d sources, Makefile, css, ECMAScript, LilyPond macros) is independent from this application and could be reused off the shelf, or with target-specific modifications.
41The pros and cons of this approach are:
-
The abstract markup structure of multiNotes, and the concrete workflow of editing, compiling, and control, could easily be taught to and understood by scholars even with little or no programming experience. (Only a small part of the markup inherited from the general purpose article/webpage definition need be considered.)
-
- 14 The empirical study by Novakowski and Hadjakos (2023) shows that participants used to very differen (...)
On the other hand, the use of a compiler solution like LilyPond for note setting is obviously a matter of taste, experience, or fundamental learning style.14 Of course the interspersed music segments can be produced by some graphic and interactive note-setting tool. For the author of the Mahler study it was important to avoid any media break. Thus only one large source file could contain to-do lists, early sketches, alternatives, and comments, serving finally as the source of the publication. Free editing (including consistent search and replace, auto correction, and statistics) was always possible without technical interference, especially that which crossed the borders between graphics, music notation, and discursive text.
-
In any case, PostScript for generating the graphic additions to the note layer has turned out to be convenient to use and easy to teach—especially since only a very limited subset of the whole language is required.
-
On the other hand, cases of complicated language stacking, as depicted in figure 5, require a skilled programmer. (In spite of forty years of professional programming experience, the author found it tedious to write this code. And even more tedious to debug it.) Further abstractions would be required to make such features useful for domain experts.
-
A disadvantage of LilyPond is that it is not very stable: using a new major software release often requires adjustments of the sources. These can partly be automated for pure LilyPond sources, but a multiNotes processing chain must always include the correct release of that software. (Luckily, these have always been provided by the LilyPond home page.)
42Some possible improvements of general interest have already been identified. On the large scale these are:
-
A syntax-controlled editor, either for d2d in general or for the specific syntax of multiNotes, would be very helpful. In principle a generic editor is easy to implement when it can run in parallel with a d2d interpreter. The authors started on an Emacs-based implementation, but gave up due to lack of time. A presentation of the intermediate XML, highlighting different types of elements, would be easier to implement and also helpful for debugging.
-
D2d has a sophisticated mechanism for supporting different human languages. This is not yet used by multiNotes.
-
Some checks (e.g., for duplicate identifiers) that are currently left to the subsequent XML processing should be performed explicitly, to support less-experienced users.
-
Let the scrolling by ECMAScript generate history entries, so that the browser’s Back button will work. (Not very urgent, because in most cases the navigation chain does the same.)
-
In texts that treat multiple movements or works, the identification of all excerpts only by measure number will not work.
-
Since both a d2d text itself and its type definition are just another encoding of XML, the backends of the multiNotes rendering pipeline could be switched from direct XHTML plus LilyPond plus PostScript to something more abstract, such as TEI plus MEI plus SVG, or similar.
-
Motif identifiers occurring in the analytical additives to music (see the green brackets in figure figure 4should be clickable (anchored), or even part of the navigation chain. This is not trivial, because geometric areas of the generated .png file must be made active in both roles, as receivers of click events and as jump targets. This task should be much more naturally solved with an SVG/MEI backend.
-
In the spirit of “pure phenomenological analysis” it would be much more appropriate to arrange the note segments exactly the other way: to re-print the exposition as a comment to the music in the recapitulation section. Currently it is the other way round, which can be considered a design flaw. For a correction, only the XSLT code need be adjusted, not the d2d/LilyPond sources.
43On the smaller scale, the following shortcomings have been identified:
-
Searching for measure number references with a search target explicitly entered by the reader.
-
Navigation chains could also be useful for citations.
-
More restrictive character parsers, e.g., for ta (= “Tonart” = “tonal key”) or Hochton (= “local maximal pitch”). These would prevent inconsistencies, as they can be found in the original state of (Lepper, 2015).
-
The inverse of the ZUK tag, as in The only #ZUK places are #NZUK place is #/ here in measure 12#ZUK and later in the recapitulation section#/.
-
Not only diatonic keys but also mere pitch classes should be modeled as full-fledged entities, with registers and navigation chain, etc., as it is done for both by Lepper (2023).
-
The same holds for single harmonic symbols. But how to treat sequences of these is an open question.
-
The arrow symbols arrn, arrs, arrne, and arrse, rendered as ↑, ↓, ↗, and ↘, have been used as abbreviations for upward direction, etc. These mere graphically defined symbols should be replaced by some more semantic markup.
-
Playing sound samples when clicking on the notation excerpts could be sensible—even more so for clicks on a motif identifier.
44MultiNotes and d2d are freely available as part of BandM metatools, the authors’ compiler construction tool kit. Source text publication on a hosting platform is in progress.
SVN keywords: $Id: jtei-lepper-253-source.xml 1267 2025-01-12 13:38:18Z pietro.liuzzo $