1While the Text Encoding Initiative (TEI) community develops and distributes several TEI-specific tools,3 there are a host of open-source tools created for more general uses that can be used to make and enhance XML that follows the TEI schema. One such tool is OpenRefine, a “powerful free, open source tool for working with messy data.”4 It was first developed in 2010 by David Huynh as the successor to Freebase Gridworks, and named “Google Refine.” In 2012, Google transferred ownership and maintenance of the tool to the open-source community and it was renamed “OpenRefine.”5
2The tool has attracted a particular type of audience in its fifteen years of existence. In a 2020 user survey, 37.64% of respondents identified as librarians. Librarians continued to be the most significant user group of OpenRefine in the 2022 survey, with 15.1% of users identifying with this category.6 In 2024, the most prevalent self-identified user group was “Researcher” (38.50%), with “Librarian” coming in a close second (34.01%).7 Note that respondents to these surveys can select multiple identities.
3Which tools individuals are familiar with is influenced by their education and professional circles. I attribute my own knowledge of OpenRefine’s existence to my graduate education in information science and frequent workshops and presentations on the tool in subsequent professional conferences that I attend in my professional capacity as a metadata librarian. OpenRefine’s capabilities are well known and frequently shared within the library profession, but mention of the tool is less pervasive in other communities. Upon first becoming involved in TEI creation through the Maria Edgeworth Letters Project (MELP), I did not see any sources suggesting OpenRefine as a tool. Nonetheless, I decided to use OpenRefine for specific use cases within the project, as it is a tool I am comfortable with, and it has decreased the technical barriers to participation in the project through its reliance on spreadsheets for data entry rather than the creation of XML from scratch.
4I have since become aware of other TEI projects that use OpenRefine, but mention of the tool in TEI workflows is still not common. One such project is Mapping Victorian Literary Sociability, a grant-funded initiative developed by the University of Calgary.8 As with MELP, the introduction of OpenRefine to workflows in this project came from a metadata librarian, Ingrid Reiche. In the project, spreadsheets were transformed into TEI using the GoogleRefine Expression Language (GREL), as they are also in MELP’s workflows (Bourrier et al. 2021, 307).
5OpenRefine is a unique asset that the TEI community should seriously consider regularly integrating into its workflows because of its openness and its affordances. Unlike oXygen, a proprietary program developed by SyncRO Soft Ltd. that is at the center of many TEI projects, OpenRefine is accessible for free. Because it is an open-source project that is continually evolving, it also welcomes enhancements to its functionality by those using it, including the TEI community.9 In addition to its potential for developing new functions, the tool’s current affordances make it extremely applicable to TEI projects. Its ability to generate TEI from spreadsheets using templates and to reconcile entities with authorities will be illustrated using the Maria Edgeworth Letters Project as a case study.
6MELP aims to make an open-access digital archive containing the complete correspondence of the Anglo-Irish Regency author Maria Edgeworth (1768–1849) and her circle. The corpus consists of approximately 10,000 extant letters held by 40 institutions. Edgeworth was a contemporary of the novelists Jane Austen and Sir Walter Scott. While she is not as well known as these figures today, she is considered the most commercially successful novelist of her time (McCormack 2004). Her Tales of Fashionable Life, a conduct guide, is particularly known for the revenue it generated. She is critically noted for her novels Castle Rackrent (1800) and Belinda (1801). In addition to her novels, Edgeworth is noteworthy as a children’s author and educational theorist. Collections of children’s stories by Edgeworth include Moral Tales (1801) and Popular Tales (1804). Her letters also provide evidence of her interest in and contributions to educational institutions locally in Ireland and abroad. As the second oldest of Richard Lovell Edgeworth’s twenty-two children (Butler 1972, 7), Maria Edgeworth’s interest in childhood and education may come as no surprise. The size of her family is also significant for this project, as Edgeworth corresponded with her siblings regularly, and these letters provide information on family relationships in the early nineteenth century.
7MELP is an interinstitutional initiative, and currently includes members from Texas A&M University, the University of Tennessee at Knoxville, Southern Methodist University, Wake Forest University, and Xavier University of Louisiana, an HBCU. The team consists of five general editors, a project manager, and a technical editor. As a metadata librarian, my own role has focused on TEI creation and workflows. The team was awarded a National Endowment for the Humanities Foundations–level grant for the years 2022 to 2024 to achieve its goal. In addition to the team members, a large number of students and volunteers have contributed substantially to the project. Graduate students are primarily responsible for encoding transcribed letters into TEI and for completing the research needed to reconcile persons, places, and works mentioned within the letters with established authorities. A central goal of the project is to equip English students with technical knowledge and digital humanities skills.
8The work of these many contributors has made it possible to complete the encoding of more than two hundred letters and launch a beta site that makes these letters discoverable.10 Texas A&M University is responsible for providing the infrastructure that makes this site possible. WordPress serves as the front-end website, while the data are stored in a database called Corpora.11 As the landing page emphasizes (figure 1), in addition to searching the transcribed text and metadata via “Search Letters,” visitors to the site can also browse the collection by “People,” “Places,” and “Works.” These lists of entities are established in the project’s personography, placeography, and workography files, respectively. OpenRefine was first introduced to the project while establishing the workflow for the creation of these indices or “ography” files.12 More recently, OpenRefine has also been used to create minimal TEI files that will ensure that all letters the team has digitized are discoverable via the website, even when transcription and encoding is not complete.
Figure 1. Landing page of the Maria Edgeworth Letters Project beta website.
9Whereas the core of a TEI document lies within the <body> element and encompasses the original text of a document, OpenRefine is most useful in TEI workflows outside of this element, where the document follows predictable patterns. Its major applications include matching named entities with authorities and wrapping metadata (rather than free text) with the appropriate TEI elements. Data entry within the <teiHeader> as well as the creation of indices of concepts, like a personography or placeography, can be completed quickly and accurately with OpenRefine because this work is characterized by predictable textual values and XML structures. The use of OpenRefine also allows the project to broaden its contributor base by including those who may have no previous experience with TEI or XML generally. While MELP’s workflow involves graduate students using a TEI template file13 to enter metadata and manuscript text into the XML directly using oXygen, students gain this level of expertise only after having worked on the project for a few semesters. Many students are able to work on the project only for a semester, so having processes that involve more basic data entry into spreadsheets is a practical way to make use of their content knowledge. These processes also allow meaningful participation by undergraduate students.
10One of the first contributions many team members make to the project is identifying and defining named persons, places, and (literary) works mentioned in the letters. To support the browsing categories highlighted on the landing page of MELP’s website, these named entities all need to be defined in indices or “ography” files. This work is often done while reviewing text files of the letters, which have been transcribed through crowdsourcing on Zooniverse,14 though the addition of new entities can happen at several points in our workflow and is very iterative in nature.15 Each letter has been transcribed by three individuals, so the transcripts require review to resolve conflicts and ensure accuracy. These transcripts are saved as unstructured text within .txt files at this stage, so the work of identifying entities typically precedes the creation of TEI. Positioning this task at this point in the workflow makes it accessible to novices to TEI or structured text generally. When a person, place, or work is mentioned, the team member consults the appropriate master spreadsheet to see whether that entity has already been defined, or needs to be added. If it is not present, the team member researches the entity to see if it is established as an authority in the three sources chosen for this project. For persons, the Virtual International Authority File (VIAF) is consulted.16 GeoNames is used to define places.17 Literary works are established using Wikidata.18 If a match is found, the controlled string value is entered in the spreadsheet. Guidance is provided on how to search these authorities specifically, as is general advice on avoiding false positives.19
11While these spreadsheets are continually updated, a parallel process is also occurring of periodically transforming this tabular data into TEI using OpenRefine. This process primarily relies on OpenRefine’s “Templating” feature, in which the GoogleRefine Expression Language (GREL) is used to create export templates that insert the data within the appropriate TEI elements. Export templates for MELP’s personography, placeography, and workography files can be found on the project’s TEI GitHub repository.20 Example code along with the output in OpenRefine is also given in figure 2.
Figure 2. Templating export dialogue box in OpenRefine, showing GREL code and output for the creation of the personography.
12Beyond its key function of transforming flat data into hierarchical XML, OpenRefine is essential in cleaning and enhancing the spreadsheet data. Many common data-entry errors can be identified and eliminated using OpenRefine. In MELP’s workflow, a check is run to ensure that all @xml:id values are unique and that every term has an identifier, using the “Text filter” feature. Additionally, any unintentional spaces in the names of the entities are removed using the “Trim leading and trailing whitespaces” function. OpenRefine’s ability to reconcile string values with established authorities also makes it an excellent tool for the creation of “ography” files. Rather than create detailed prosopography files from scratch, the MELP team reuses data in existing authorities and contributes to these authorities when appropriate. The authorities MELP chose all have existing reconciliation services established in OpenRefine, which was a factor in their selection. For those wishing to contribute to Linked Open Data (LOD), reconciliation services are beneficial as they make it possible to automatically add a URI to an authority based on a string value. Some services, such as the one for GeoNames, also allow inserting additional information such as coordinates into an OpenRefine project. OpenRefine comes out of the box with the Wikidata reconciliation service, and other services can be added as desired. Jeff Chiu has created a VIAF OpenRefine reconciliation service, and instructions on its use are posted publicly online.21 The University of Tennessee at Knoxville currently hosts a reconciliation service for GeoNames,22 which employs code created by Christina Harlow.23 Although reconciliation services can be sensitive to slight variations in string values, when data entry is consistent with expected values these services can save a great deal of time in matching strings with their associated URIs.
13Complete instructions for cleaning data in OpenRefine and transforming the data to XML can be found on MELP’s GitHub.24 While I, as the metadata lead on the project, was responsible for creating the GREL export template, anyone on the team can update the various “ography” files by following the instructions. Some technical knowledge is required initially to establish a process like this, but the sustainability of the project is not reliant on it.
14As the team continues to work on this project, we are committed to making all of Maria Edgeworth’s correspondence available in a timely manner. While fully encoding letters containing disambiguated entities is our ultimate goal, data this rich takes a long time to produce, which is a barrier to access. The solution the team has decided upon is creating minimal TEI files that include basic metadata in the <teiHeader>, as well as references to the digitized pages so that they will appear on the website. This will allow all the letters to be discoverable and viewable as an open-access resource to the public before they are all fully encoded and edited. This follows the theory of “more product, less process” (MPLP) introduced formally to archives in 2005 (Greene and Meissner). This theory privileges access to materials over arrangement and description of each item within a collection, and promotes simplifying workflows to essential tasks in initial processing. In addition to making Maria Edgeworth’s full epistolary corpus more accessible, this approach has the added advantage of eliminating human error when entering data directly into XML files using oXygen.
15MELP has begun using OpenRefine to create TEI files from existing spreadsheets of minimal metadata. These spreadsheets include a letter’s filename, title, date, sender, recipient, and the institution that holds the physical letter. This minimal metadata was initially created so that letters could be submitted to Zooniverse to crowdsource transcription, so it did not require additional effort. Beyond meeting Zooniverse requirements, this metadata also maps to many of the elements present in the TEI header, including <title>, <author>, <persName> values within <correspDesc>, and <orgName>. Within the <body> of the TEI, filenames are placed within <pb> elements and the letter’s date is added to <date>. An OpenRefine export template is used to realize this mapping by nesting the values found in the spreadsheets within the appropriate TEI elements. The GREL needed to achieve this can be found on MELP’s TEI GitHub repository.25 Information that is consistent across the TEI files, such as the <editorialDecl> and <principal> statements, is also entered following the standards set by the project’s TEI template. The result is that a basic TEI file is created and is ready for full encoding (figure 3) in the future. Comments are present within these files, as they are within the standard TEI template, to help guide contributors in entering text consistently in the appropriate locations.
Figure 3. Selected portion of a minimal TEI file.
16Once added to the project’s GitHub repository and recognized by the Corpora database, these files allow scans of letters to appear on the beta website along with metadata. Comparison of a minimal TEI file (figure 4) with a fully encoded one (figure 5) shows significant differences, but both provide access to the scans. One of the key goals of MELP is to make all of Edgeworth’s correspondence discoverable, as no existing edited volume of her letters currently achieves this, so it is crucial that all letters are retrievable on the site. At present, the transcription box for these letters remains blank beyond the inclusion of the letter’s date. Additional work considering the best way to share these letters with minimal metadata in context with the fully encoded letters will be pursued in the future.
Figure 5. Letter represented by a completed TEI file displaying entities defined in placeography. https://mariaedgeworth.org/letters/Edgeworth_Huntington_HM28606_1826_11_08.xml/.
17As a generalist tool for data cleaning and transformation, OpenRefine has a strong user base within the library community and the potential for more applications in TEI projects. The Maria Edgeworth Letters Project has found that OpenRefine has been effective in opening up TEI creation to students, by minimizing technical barriers and producing minimal TEI files rapidly. Its ability to support reconciliation services and minimize human error also make it relevant to TEI projects. Arguably, the adoption of generalist tools like OpenRefine by the TEI community also has the potential to broaden the type of contributors participating in TEI projects, which in turn could reciprocally bolster awareness and use of TEI.
18After completing the Foundations NEH grant, the MELP team is excited to pursue a second NEH grant at the implementation level from 2025–2027, to complete the work necessary to share all of Edgeworth’s letters.26 Its primary goals include fully encoding at least an additional two hundred letters in the next three years and making all of the remaining letters available to the public using minimal metadata created by OpenRefine.
SVN keywords: $Id: jtei-hale-285-source.xml 1338 2025-11-19 15:42:27Z ron $