Zwischenheit. For soundscapes, audio captioner, and local language model

Contemporary neural audio research frames “music understanding” as a computational task. What does it mean for a machine to listen and understand a sonic context? Zwischenheit (2025) is an audiovisual performance that aims at finding a speculative, empirical, situated answer. The projection shows the performer having an improvisational dialogue with an algorithmic system composed of an audio captioner and a local language model. While the sound piece unfolds, it reveals a complex scenario made of overlapping soundscapes. The language model is prompted to interpret the music as it flows, trying to provide a nuanced understanding of the sonic situation. The human performer, on the other hand, is as inquisitive as cogitating: at which threshold does the language model begin to appear as an agent of mystification? What does agency without consciousness reveal about listening? The outcomes of the dialogue change at every performance, as there is a certain degree of stochasticity in the model’s replies, but they always point at critical aspects of sonic hermeneutics and computational cognition. Embodiment, contingency, and situatedness emerge as essential characteristics of human listening that contemporary neural networks cannot embed. Zwischenheit is thus an attempt at investigating the performative possibilities that emerge at the intersection between post-acousmatic music, music information retrieval, and generative AI through an analytical self-reflection.