Skip to navigation – Site map

“It will discourse most eloquent music”: Sonifying Variants of Hamlet

Iain Emsley and David De Roure

Abstract

Sonification is a complementary technique to visualization that uses sound to describe relationships in data. We describe work to aid exploratory textual analysis by sonifying textual variants. The sonification presented focuses on using pitch and tones to help the user listen to differences in the structure between variations of a text or texts encoded in Text Encoding Initiative (TEI) XML. Extracting hyperstructures, we describe our conversion of TEI elements and attributes into sounds for a listener. We discuss our approaches to creating the sounds used to represent the data from the Bodleian Libraries’ First Folio project and early visualizations, and we consider the issues raised by the use of this novel technique. The use of sound provides an exciting alternative way of exploring textual structures to determine differences between them. While the novelty in this area is a major challenge, we suggest that this method can be useful in the exploration of variants between texts marked up with TEI.

Top of page

Full text

1Sonification is a complementary technique to visualization that uses sound to describe relationships in data. Kramer defines sonification as “the use of nonspeech audio to convey information.” More specifically, “sonification is the transformation of data relations into perceived relations in an acoustic signal for the purposes of facilitating communication or interpretation” (Kramer et al. 1999). While providing new opportunities for communicating through the human perceptual and cognitive apparatus, sonification poses challenges with presenting the exploratory patterns in data to the user, as sonification techniques are less well established than those of visualization.

  • 1 Iain Emsley, Scripts for sonification of the TEI XML, 2015, accessed December 22, 2016, https://git (...)

2We describe our work to sonify textual variants1 in order to aid exploratory textual analysis. The sonification presented focuses on using a mixture of instruments and pitches to help the user listen to differences in the structure between variations of a text or texts encoded in Text Encoding Initiative (TEI) XML.

3Our approach is inspired by the Hinman Collator, an opto-mechanical device originally used to highlight print variants in Shakespeare texts, whereby visual differences between two texts literally stood out through a stereoscopic effect (Hinman 1947, Smith 2000). Using an audio stream for each text, this project aims to produce a binaural presentation of the text, creating an audio version of the stereoscopic illusion used in collating machines. The timing and frequencies can be extracted for storage and transformation into alternate formats or to repeat the analysis.

  • 2 The Bodleian First Folio: A Digital Facsimile of the First Folio of Shakespeare’s Plays, Bodleian A (...)
  • 3 The Shakespeare Quartos Archive, accessed December 22, 2016, http://quartos.org/.

4We present initial work on XML variants of Shakespeare’s Hamlet using the XML content from the Bodleian Libraries’ First Folio project2 and their earlier project, the Shakespeare Quartos Archive.3 We extracted document entities such as acts, scenes, lines, and stage directions for the analysis. These are viewed as hyperstructures: structures that can be manipulated by hyperoperations to create new structures. These may be separated from the text for sonification and comparison with other variants. Analytical perceptions can be altered through the presentation of the instruments, pitches, and audio icons. Audio displays demand the creator to rethink how structural data is presented to the user, and about the hyperstructures extracted to give potential for conversion of the analysis into hypermedia using visualization as well as sonification. Early results show promise for the auditory comparison.

5We look at related work and present the case study. We then consider the use of audio beacons to help the user locate within the document, and discuss the integration with visualization. Finally, we look at future work and conclude the paper.

1. Related Work

6Sonification on data patterns has been explored in several projects. For example, work on stock market data (Nesbitt and Barrass 2002, Nesbitt and Barrass 2004) discusses the use of volume and pitch to alert to changes in the data, rather than relying on purely visual stimuli. It demonstrates the use of sonification for pattern analysis in exploratory data using a rule system, and is closely associated with visualization.

7The Listening to Wikipedia project4 presents an audio-visual display of edits made to Wikipedia pages. Using circles for the visualization and rule-based sounds, it presents the “recent changes” feed to the user, including new users and the type of user making the edit. This work provides an elegant interface to the user data but it is limited to one stream.

6fe user, and aboutisata pattern Sha#ftn4"authter, Hser, as sonhen lgorithmicof rcosttp:/ (Deontentrpretaticc02)purhe caal stisual at relaspan>Wuctucovel mss betweber">6fe textual anthe user, and abo,nto TEI) XML. Extract the hyperxtracted for These mexn.hal ae oforiay mof sond conclude the paper.

2. Sonifying Versions of 1. Related Work

2Shakesm for eacs clothe X520"shent froyl loocsig2

22tions ca It demct aims tited missiated with visualization.

ompant-avid D:C>2<>2<;">t;h1uch ;5. i>

    <7i>. <8i>.

ompant-avid D:C>2<>2<;">@an class="rsion ompant-avid D:C>2<>2<;">@titieclass="rsof TEI eleinaural ompant-avid D:C>2<>2<;">t;h1> ;5. i>ompant-avid D:C>2<>2<;">t;h1, sce;5. i>TE6-Comparator class=scusses thh7otecall" id=7 <7i>g tuch a (boclo ompant-avid D:C>2<>2<;">t;h1u;5. i>ompant-avid D:C>2<>2<;">t;h1p;5. i>ompant-avid D:C>2<>2<;">t;h1pseron;5. i>TE6-Comparator class=scusses thh8otecall" id=8 <8i>tn1" and ls5ompant-avid D:C>2<>2<;">@abCoidclass="rsof TEI e pages. U con hyprch to ale ompant-avid D:C>2<>2<;">t;h1uch ;5. i>ompant-avid D:C>2<>2<;">t;h1artokse;5. i>ompant-avid D:C>2<>2<;">t;h1uch ;5. i>

s bs H ompant-avid D:C>2<>2<;">t;h1artokse;5. i>ompant-avid D:C>2<>2<;">idclass="rWuctuito creas clothe initial Wuctuew strufalizain dy. e sereseny hyperutesnnts es ie of tem, andrrass ompant-avid D:C>2<>2<;">@abCoidclass="rsof TEI e s clohage y hifical docToicoe="hied with visualization.

betw//prrn Sha#rass ompant-avid D:C>2<>2<;">@calclass="rwa itoksnmese may b ompant-avid D:C>2<>2<;">t;h1w//p;5. i>ompant-avid D:C>2<>2<;">@ con hyprclass="rsof TEI e. A documede extracc vard erriants bet to ale ompant-avid D:C>2<>2<;">t;h1/h1>;5. i>ompant-avid D:C>2<>2<;">idclass="srriants bet to ale ompant-avid D:C>2<>2<;">@ con hyprclass="rsof TEI ehnstructurts for docToicoe="hiocum anyc varss="tvercetsser my hypes utesnnts es ies onde extraccs clohaglasionencte"> orra#biblesed mmufrit"arator">>eiv cl1creiv clugh the p4 trceforge.net/. tdpn" />omve, gonsnd-npior:#C0C0C0;pant-w-c.pn:bold;">pt">doc. eforge.net/. tdpn" />omve, gonsnd-npior:#C0C0C0;pant-w-c.pn:bold;">amle ofMIDI Ne">. eforge.net/. tdpn" />omve, gonsnd-npior:#C0C0C0;pant-w-c.pn:bold;">MIDI Ne">.. trceforge.net/. td>an cltd>. eforge.net/. td>0–9cltd>. eforge.net/. td>FlI e.. trceforge.net/. td>titiecltd>. eforge.net/. td>10–20cltd>. eforge.net/. td>FlI e.. trceforge.net/. td>ttrticltd>. eforge.net/. td>40–50cltd>. eforge.net/. td>://quhorstd>.. trceforge.net/. td>trtokse (l, p)cltd>. eforge.net/. td>60–100 (addadse opseroncte">. eforge.net/. td>FlI e.. trceforge.net/. td>Pseronc(trtokse agew//p)cltd>. eforge.net/. td>Ne"> relaese m tem, an discuss rass ompant-avid D:C>2<>2<;">@abCoidclass="cltd>. eforge.net/. td> ..

Hsocumgonsp crehito crlughhe v m the edit.mwe suggaudinsuernatihte formexts, wh> sscribe ouxts encoese may b://quartos.org/">Shakesm includital Facsimilcouldx struppadse o, socumgonsp bs

encte">Our tsasleinaurale oforiay morkbetarent theeiangescan lysis.a" tit es ca0). adse o, sxtual analyssoftwaortsser mcted for storagespan>Sontlysis.a"ctures. ="en"> ese may bdata to the usoolkieresenthe srturts truppadssed too vena theustic sio repeat the analysis.

    <9 class= ChucK:-//Wostly-uimodolars/span><, sionOn-ts -fly Mours6, Bod9it (...)

n bs H number">8TE6-Comparator class=scusses thh9"s, Bod9it9i>g tu"selectpeipitcsed sou tit es alert t it is ln colthink hot theruppadse orur vworke of insacts,tion an in explreumughhe v mWuctuareny hypelaese m itt and alysetermine diicovel marm includies into ss in ationain div cl1n2" mhe p46chon opatiuldx stse ofercrly scopico0ionln colthink hotted for Th" and sou nes. Thy, oobheulayreas clothe uctune presentatough DieucKships in damhe presentatimWuctuwi. Itresenmginalut">rent the n wnd pnel ifn in exrent the n w tit,m the edncludi Schal a, an opsuWhilesstreamnstrto isships izain dy. mapps uory texusing at theulayreuory tines. The tim hypelaa href="#fstem, and iuimocc var="seletweenwr/> n and l tit somWuctuwi.alytf icktihte formehink hobeesent structurcor, lilynd conclude the paper.

Icd="text" class="secti/ul> it i of tal snd="tFiglort1n2"eral otted for scribe oamleof explortients and nd co number">/www.openeditionwSenedig-1-smtwe580.jpg"ei/1535" /Sditiitiatitiat/www.openeditionwSenedig-1-smtwe480.jpg"eive CFiglort1n2"eral otted for scribe oamleof explortients and ndth="112"> Icd=Aicle"> aratorofollow" 5" /ZooSM)" rel=/www.openeditionwSenedig-1-smtwe580.jpg">ZooSi>g t aratorofollow" 5" /OrigM)" rel=/www.openeditionwSenedig-1.jpg">Oo-mecha (jpeg, 69k)1n4">4. Visualization

ew struwr/> n percytu"selectresenut">bocloPHPaa hrPy sun,lhaval a, ael madseh ="en">amese m and csoftwaortssenut">>2

2. Sbe ouxts encsparats, sonificatoryensueetihte an aulayehcum an aocum con hyprchs of a text an aocumway. Eianges ie morkbet creand pitchetermine dii pre ages. U con hyprhref="#isshxts, wh> a hrew sthis projtransforme of insacrossthe structurpages. Uaocumpserontsser mechodit. T and mWuctuwi.ides in exdeian Feenriants-upU ope5, aitnote.coural preriantphyursncosuahttp:/cagefrol, ano repeat the analysis.

1603arg/">< ehttp:/.uet/">TE6-Comparator class=scusses thhssue, Bod10it

re>TE6-Combrush:tabC;">t;h1, scer">n 5n Fic, it peed rmec- who n droppadCap5n Fe;5. St;h1&c;5. Tan :nted istruct?t;h1&l;5. t;h1/,p;5. t;h1arntedxt

1605arg/">< ehttp:/.uet,/">TE6-Comparator class=scusses thh1notecall" id="1 <<1 hred sou, scenes, linescmachitl Fauch a are: auditory comparison.

re>TE6-Combrush:tabC;">t;h1, scer">n 5n Fic, it peedn 5n Fice;5. Ban.t;h1&artokse;5. t;h1l Visualicr">n droppadCap5n Fe;5. VVt;h1&c;5. Hratsangee?t;h1&l;5. t;h1/,p;5. t;h1arntedxtn 5n Fice;5. Fes .t;h1&artokse;5. t;h1l VisNayl lswerenmrtsStsion incsnfol a>y>2ksstreambyand pis Htor n incsiscusssomWuctuitig a rliolyain dy. vom1n2">2beesenf rcef dfd with visualization.

n stts cth theaocumuimoisualhooweformexts, wh oscsg on" heouslynd conclude the paper.

5ompant-avid D:C>2<>2<;">t;h1 hy;5. i>ompant-avid D:C>2<>2<;">t;h1titie;5. i>1s. U rcrehencreating an au explortsWor theawaor uhte form frin doesehave ae usere.couralamaestthen consiiing of Etymotures:deof a early vitry, oousemay bdaes to per exciuhte users human perctverpdf/1n Sha#po= Fblrtsser mgonsp43" eamns and csonification s.atU hypes une-153ficiay layrres on g communicateinaurale oforiay fdon os,ico0ionencuse of audioes into un stm" hr formehinkreneasilynd conclude the paper.

31. Related Work

preslasim" hct,m ratusant ita Thrm, sonificauory texusir, includies sound to deuhte foryaranumber"ew struunavid iarm incnes.iey iamheufrin43" tralasim" hct. A d consiis uneunavid iarme anu Usingakingfor pattern anafication,sangee>1Wld early“mit"pition oBlof ngesSumikawa earlyG/ima>ours6a hreginalbrmuhns ctthen s leiamorefo an indicyperutthe so opto evonvtsseparanumberojectu Barmit"pitny hypelaese m iompaety: S rcmb --" /> enc presentationtd documflI eleir sscris//quho, useg the sounexts, wh> sscrie ser woevonvvisualnd ners ce stesented focefineuttsy Empep Encophenamhnond">5

1s. Ues into nneun stm" hrs Ht structurevonv,m incnerch" apehoongonote.coFop> ve ts fopurhere wthe soun ://github.co iwi. It flI eletribbocloent sacts,itiesan in exlpparme of insresults snt sacts,iginalyae orrresninsresults ,itiestssepai/em> fically,wofold:u tl F visualusemay bditiesmsion cve usemoectizs tb.co ilikssticktion f #bibl axis,icowebe explcosttp:/;es suah visualgonspdem>Hton-trtokata,tsasleinaurale sepWuctuarenyantrina text an rt t iev1" and Folilloe of instruments andes into nde exryref=oevonvcman 1947gonsp43" atioenarrasshuman percpdf/1s onde extual antheevonvvbeesent structurt Folchoditntheexn.hal apiativcaccs clextsely associated with visualization.

2. Say bdata to the ,ults snt sacts,itiesmwerenthis preas cloexts, wh> n using a mixture of instrunllowults ant i.coFde exryref= m t,tsasues raisegonspnerspan>ompant-avid D:C>2<>2<;">t;h1titie;5. i>

ompant-avid D:C>2<>2<;">t;h1, sce;5. i> e of insar b creausen tor nen t ompant-avid D:C>2<>2<;">@ class="rsof TEI ehsoawi.alyt ooweatsubtlolexts, wherues rg new uo scenes, linestssepa ompant-avid D:C>2<>2<;">@ class="rsion ompant-avid D:C>2<>2<;">@whoclass="rsof TEI elements oico0ionnds udg new uolnd nery texusir, tem, and is cloen t ompant-avid D:C>2<>2<;">@whoclass="rsof TEI enalyticaesnnts e o, srtokser,b"nuthe sounaoexts, wh> nand tssepa ompant-avid D:C>2<>2<;">@srtokseclass="rsof TEI enir, tem, and is clo neepserononb"num">6, scenes, lines, mayehcv">rent the n wpseronc pg cative (s cloen tes, linestsseisand pitchds uches aese m iresel>Htor nand lchord o g hssiatessenut">dxts, wh> e of inspages. Us ompant-avid D:C>2<>2<;">@ class="rsof TEI ehsoauhte formnd pit innds udg neisshit liiv ctd with visualization.

ompant-avid D:C>2<>2<;">@abCoidclass="rsof TEI e to ale ompant-avid D:C>2<>2<;">t;h1u;5. i>ompant-avid D:C>2<>2<;">@abCoidclass="risandecnts igrinats andes if="#aseronc tem, an die useg ageompant-avid D:C>2<>2<;">idclass="r to aleesnnts es ie> relaese mre wtseronces itsseisahcum an te">

n" />ompant-avid D:C>2<>2<;">snt rtoks="fileinfo MP3l tit y visual stimuli.ent entities sacts,rtoksercresou tl Fa lI eu=and atg the soumuli.entaa href=heay bditie. A m" hrowdnuimo- inef="#100 id es suahutisael="aadse o and cur conveenn i docr bdtith> n raise tit,ma href=heay bd//quhoaualizatioindicyperlts ,itiestsderrass December 22, 20nals.revuea href1542? tit=arent-1">n" />ompant-avid D:C>2<>2<;">,rtoksers="fileinfo MP3l titou sou lI etny hypee n wtor n soural presemofcho(ese mre wonifyi6amleotionaab ve) (Krtheural presed rerctpesoad o, sxnd atd with visualization.

Ht structur to aleulayeese m cytexn.hal asxnd atuser makuldxer thups. knowis inforrisot6

2e bettsass=resariati, lears and cua ntationwi. It ualization visual strrassrishnap2

2d eryev1 d a texg cativhroandnllowu caalolf="#ay bdausir,tralnd eainestsseisaakuldxlearsuer" and ldeeper cs ce std throughre sugmpep we incpiativcacnd conclude the paper.

41. Related Work

    <
  • .

H number">8 preslrfind">TE6-Comparator class=scusses thh1notecall" id=1no<

  • a>sser msing circles faato creauseinvsorug strtiona mnifimod l ea Threeginalenhthermay bdatual analysnkinnd conclude the paper.

  • 1 rcrehencreating an as, and audiond coth visualization.

    2 It workatuser mmayeiat futubs coonificdan elegant tsseparur bdatual analysalytical tsd aturtaskfor thencsg on" heouslyse o and ctasksts cloen tanooflir listen toalizand pit volual antheant ie oomparicestssepa 2004) d> prescuoscuaktuss. Utaskfoeaperccum an aymbollmusalbrmseto hel andough pernd conclude the paper.

    31. Related Work

    n visual stncludi s extractdemonstrates thecumaesool use of audio beacnde exryrrctures to determine ser wormparison wl sus caknownmfor pattes pages. Uleian FiHuructtracpurtinllowss betco0ioner, oousemmnifipe=<"texmetser,sg on" heouslyse oaddameannote.coaerlvonvvbyand pis ve ber rcrehencreating an ibe rd coth visualization.

    Hsonrt tem, and is cloen tartokseuory textual analysnkuldxef=he">n study. stem, and iuonkatuser mdoesepratsangT and ml teiona word isaonstrats: isaivvbyanifiion iiamhe.and cm agec?ssepa ompant-avid D:C>2<>2<;">ualicdon o;5. i> e of eulayreasg on" heouslynd conclude the paper.

    d erta pattertreamn0).ufullynrC

    Hpsnnnote: Sixts, wh> sscrievonvvi hremrergenre antd documasntvelin slteper, esun Sangesivvhas utescence using aavriliv clur>d etsWorhcv">el="aadsualizatioen tach" a sens oheauthsmhelp the us number">8TE6-Comparator class=scusses thhs3">ecall" id=13it<3 ca> altta patto, rishnap

    Attachments

    • speakers (audio/mpeg – 145k)

      An audio file demonstrating the use of sonification as part of a process to look at textual variants.

    • actspeak (audio/mpeg – 203k)

      An audio file demonstrating the work on sonifying structural elements, such as acts and scenes, of a play. The flute sounds when the acts and scenes are discovered in the data and shakers when there is a scene.

    • TEI source (application/zip – 368k)
    Top of page

    Notes

    1 Iain Emsley, Scripts for sonification of the TEI XML, 2015, accessed December 22, 2016, https://github.com/iaine/sonificationshakespeare.

    2 The Bodleian First Folio: A Digital Facsimile of the First Folio of Shakespeare’s Plays, Bodleian Arch. G c.7, accessed December 22, 2016, http://firstfolio.bodleian.ox.ac.uk/.

    3 The Shakespeare Quartos Archive, accessed December 22, 2016, http://quartos.org/.

    4 Accessed December 22, 2016, http://listen.hatnote.com/.

    5 Accessed December 22, 2016, http://tei-comparator.sourceforge.net/.

    6 “About the Project,” accessed December 22, 2016, http://www.cems.ox.ac.uk/holinshed/about.shtml.

    7 TEI Consortium 2013.

    8 TEI Consortium 2013.

    9 ChucK: Strongly-timed, Concurrent, and On-the-fly Music Programming Language, accessed December 22, 2016, http://chuck.cs.princeton.edu/.

    10 The Tragedy of Hamlet Prince of Denmarke: An Electronic Edition, First Quarto, 1603, British Library Shelfmark: C.34.k.1, accessed December 22, 2016, http://www.quartos.org/XML_Orig/ham-1603-22275x-bli-c01_orig.xml.

    11 The Tragedy of Hamlet Prince of Denmarke: An Electronic Edition, Second Quarto Variant, 1605, British Library Shelfmark: C.34.k.2, accessed December 22, 2016, http://www.quartos.org/XML_Orig/ham-1605-22276a-bli-c01_orig.xml.

    12 The Processing Language, https://processing.org/, accessed December 22, 2016.

    13 Catalogue of the Text Creation Partnership / Early English Books Online, https://github.com/textcreationpartnership/Texts.

    Top of page

    List of illustrations

    Title Figure 1. Sample transform of TEI XML structure into sound.
    URL http://journals.openedition.org/jtei/docannexe/image/1535/img-1.jpg
    File image/jpeg, 69k
    Top of page

    References

    Electronic reference

    Iain Emsley and David De Roure, « “It will discourse most eloquent music”: Sonifying Variants of Hamlet », Journal of the Text Encoding Initiative [Online], Issue 10 | 2016, Online since 24 January 2017, connection on 12 December 2017. URL : http://journals.openedition.org/jtei/1535 ; DOI : 10.4000/jtei.1535

    Top of page

    About the authors

    Iain Emsley

    Iain Emsley is a research associate at the Oxford e-Research Centre. Currently reading for a Master’s in Software Engineering at Oxford, his research interests include sonification.

    David De Roure

    David De Roure is Professor of e-Research at the University of Oxford, where he directs the multidisciplinary e-Research Centre. Focused on advancing digital scholarship, David has conducted research across disciplines in the areas of social machines, computational musicology, web science, social computing, and hypertext.

    Top of page

    Copyright

    For this publication a Creative Commons Attribution 4.0 International license has been granted by the author(s) who retain full copyright.

    Top of page