Aim of our Research
Objectives
Over the past decade, members of the Deaf and Hard of Hearing (DHoH) communities have made significant strides in advocating for closed captioning on streaming platforms. The decisive 2011 court case of the National Association for the Deaf against Netflix set a precedent for the industry so that today, all major services offer programming with captions that include dialogue, sound effects, and song lyrics. However, captioning for the non-lyrical, non-diegetic musical underscore - often representing the emotional core of the show - has remained mired in a patchwork of inconsistent and at times inaccurate (or absent) descriptive words.
Two issues are at the heart of the problem: 1) the score occupies a unique position within captioned soundtrack elements - it evades both the literal transcription of dialogue and the indexical identification of sound effects, and 2) the captioning of non-lyrical music cues is entrusted to the subjective judgment of uncredited, non-musician captioners.
The DHoH communities deserve to experience the full soundtrack through captions, beyond the dialogue and sound effects. “Narrating Music” intends to fill the gap by assisting captions-users and -creators in negotiating the underscore’s soundscape while coming to terms with how captioners and audiences subjectively interpret musical sounds that accompany moving images.
Through its combined practical and theoretical orientation, the project seeks to benefit multiple communities - scholars, DHoH audiences, and captioners - who are invested in how music is experienced in screen contexts.
The overall objective of “Narrating Music” is to investigate the theoretical frameworks and practical applications of captioning music to enable DHoH communities to more fully experience media. The project seeks to generate outputs that increase understanding and accessibility of music captioning by producing an affective lexicon of music descriptors that supports DHoH audiences and standardizes captioning practice.
Context
As a research project concerned with the transformation of sounds into words, “Narrating Music” draws on multiple disciplines to theorize music captioning in audiovisual media: musicology and sound studies, media and film studies, translation studies, disability and Deaf studies, and the emerging field of captioning studies. This echoes calls within critical disability studies to strengthen interdisciplinary approaches and collaboration. To integrate these perspectives, the project adopts approaches such as narratology - examining how texts tell stories - and semiotics - analyzing how texts generate meaning - while also using cultural-rhetorical analysis to situate captions within broader aesthetic and accessibility practices. Grounded in how music is communicated via closed captions, “Narrating Music” must probe beyond the act of transcription to examine the aesthetic and cultural stakes of representing music through words, while facilitating access to film and television for DHoH audiences.
Methods
The project aims to accomplish our goals through three phases, each defined by a specific objective, distributed over the course of three years: 1) collect captions from TV series on streaming platforms; 2) analyze and interpret the data to create a lexicon of affect, and 3) produce a monograph and other knowledge mobilization products based on the results of the investigation.
Throughout the project, “Narrating Music” aims to address the following foundational questions: How does the interpretive act of captioning tangibly render the rhetoric of musical sound for Deaf audiences? How is musical meaning captured and translated via the narration of written language? In other words, how do captions narrate music?
Data Sources
Virtually all streamed television programming today offers closed captions and thus could serve as primary sources for studying music captioning. “Narrating Music” examines captioning of music across Netflix, AMS, HBO Max, and Apple TV+. Among these streaming services, this project will investigate the captioning practices in full series of popular post-2011 Netflix Originals, including Stranger Things (2016-2025), The Haunting of Hill House (2018), Squid Game (2021-2025), 3 Body Problem (2024-), and Black Doves (2024-), with AMC series Breaking Bad (2008-2013) and Better Call Saul (2015-2022), HBO Max’s Westworld (2016-2022) and The Last of Us (2023-), and Apple TV+’s Severance (2022-) and Slow Horses (2022) among the comparators.
These series present the researcher with a formidable corpus of music-captioning data in order to produce the most valid and representative findings. The approach to the sources builds on methodology piloted in a previous study by Dr. James Deaville: screening each episode, noting on spreadsheets captions used for every instance of underscoring, and comparing those texts with the actual sound of the music.
Framework analysis (FA) emerges as a preferred operational method for the project’s qualitative research. As Klinger et al explains, framework analysis systematically codes homogeneous narrative datasets to reveal patterns, typically expressed in a matrix.
NVivo is a qualitative data analysis software that can be used to assist with the data extraction, synthesis, and critical appraisal of our spreadsheet entries. It offers functions consistent with the data processing goals of “Narrating Music”: entered data is grouped into cases and sets and codes are assigned to data, enabling the researcher to pose queries of the data and to create matrices that identify trends and patterns in the data. The NVivo software will be available to project researchers through Carleton University.
These tools will facilitate the work with existing captions by sorting and grouping them into meaningful clusters according to affect while identifying problematic descriptors that do not correspond to the sound of the underscore. While “Narrating Music” cannot alter or add closed captions to the streaming media under investigation, the tools we fashion can provide DHoH audiences with the means to make their own context-based understandings of a scene’s underscore, which we can confirm through the input of the Canadian Association of the Deaf.
Research Phases and Procedures
“Narrating Music” will fall into three distinct yet interrelated phases over three years: 1) the collection of captions from the series; 2) the analysis and interpretation of the data for the lexicon and matrices; and 3) the synthesis of the findings into the monograph.
Phase 1: Collection of Captions
The approach to the sources builds on the methodology successfully piloted for the case studies in Dr. James Deaville’s book chapter on closed captioning in Peak TV (2024): screening each episode, noting captions for every instance of underscoring, and comparing that text with the actual sound of the music. During this first phase, the researchers will systematically process the series, season-by-season and episode-by-episode. We will code captions and their context of visuals and other sounds to determine the prevalence and settings of descriptors, identify expressive patterns within genres, and track shows that present recurring challenges in music captioning.
Phase 2: Interpretation of Results
Applying Framework Analysis to our data through NVivo software will enable the creation of an affective lexicon of music captions, a searchable tool that clusters descriptors according to expressive category while indicating prevalence, genre/show, and screen-narrative context. Thus under the overarching register of “Uncanny,” we will group descriptors like “eerie,” “mysterious,” and “creepy.” Such clusters will assist caption users and creators alike in navigating fields of affect.
The researchers of “Narrating Music” will then map these linguistic clusters onto arousal-valence matrices for different genres, visually representing the relationships between captions. I intend that these tools will practically aid DHoH audiences in navigating closed captions for streaming television and offer guidance on consistency to caption creators. Both the lexicon and the matrices will be available on this website.
Phase 3: Synthesis of the Data into a Monograph
The data analysis will subsequently shape the project’s book-length study of the theory and practice behind closed captions for music in audiovisual media. At the same time, I intend to use our findings as a springboard for understanding how the general public hears - or reads - and interprets the underscore for moving images. The non-musician captioner perceives musical meaning even as audiences do, in an act of creative interpretation, and thus their practice of “narrating music” reflects how music “means” for screen audiences.