Adam Melvin and Brian Bridges
Cinema’s audiovisual language is often treated in terms of distinct planes of image and sound, with the soundtrack itself comprising multiple modes ranging from established musical structures to more extensively textural sound design, much of which may serve affective framing functions rather than conforming to a simple diegetic/non–diegetic dichotomy. Some authors have sought to elucidate the mechanisms and functions of how sound and image may influence each other, most notably Chion’s (1990) audiovisual contract. Whilst treating music and sound design as separate may provide for some specificity within our analytical frameworks, the acoustic circumstances of many contemporary soundtracks provide examples in which sound texture, gesture and timbre combine to delineate spatial attributes and territories. Furthermore, commentary relating to the haptic score (Mera, 2016) has highlighted imperatives around considering music and sound design in integrated contexts (Kulezic-Wilson, 2019). This paper will seek to develop the approach Mark S. Ward (2015:166) proposes for considering the cinematic soundtrack via a spatiotemporal contract based upon concepts from embodied cognition, most specifically, the image schema theory of Lakoff and Johnson (1999), placing particular emphasis on the potential for audio to articulate and structure perceptual space. It will seek to explore aspects of cross–modal integration between both music and sound design and sound/music and the image via spatiotemporal concepts and models derived from image schema theory, with a particular emphasis on embodied spatial models of timbre, elaborating upon the timbre–space models of Grey (1977) through the application of embodied concepts; see Roddy and Bridges (2018). In more integrated soundtracks, spatial perspective, via timbral articulations, can be seen to grow in importance beyond the simple delineation of mental/internal and physical/external action. Timbral gestures are seen to provide the communication channels facilitating cross–talk between axes of spatial (delineation/diffusion) and temporal (integration/disintegration) structures. As the soundtrack, in this context, is considered primarily in terms of the implications of its auditory materiality, the visual territories are reinforced or undercut with affective auditory planes whose syntax and structure seems to function in relation to models from embodied cognition. Cinema, as a multimodal, audiovisual ‘language’, is thus a contract between space and time via mechanisms and frames from embodiment. Embodied models of timbre within the soundtrack can therefore provide one means by which the soundtrack can be treated in an integrated manner, both in its own right, and in relation to the visual.
References Chion, M. (1990) trans. Gorbman. C. Audio-Vision: Sound on Screen. Columbia Univ. Press, New York Grey, J.M. (1977) Multidimensional perceptual scaling of musical timbres. Journal of the Acoustical Society of America. 61(5):1270-1277 Lakoff, G. and Johnson, M. (1999). Philosophy in the Flesh. New york: Basic Books. Kulezic-Wilson, D. (2019) Sound Design is the New Score: Theory, Aesthetics, and Erotics of the Integrated Soundtrack. Oxford: Oxford Univ. Press Mera, M. (2016) ‘Materializing Film Music’, in Cooke, M. and Ford. F. (eds) The Cambridge Companion to Film Music. Cambridge: Cambridge Univ. Press, pp.157-172 Roddy, S & Bridges, B. (2018) Sound, Ecological Affordances and Embodied Mappings in Auditory Display. In: M Filimowicz & V Tzankova (eds), New Directions in Third Wave Human–Computer Interaction. 1 edn, vol. 2, Human–Computer Interaction Series, Basel, Switzerland, pp. 231-258. https://doi.org/10.1007/978-3-319-73374-6, https://doi.org/10.1007/978-3-319-73374-6 Ward, M. (2015) ‘Art in Noise: An Embodied Simulation Account of Cinematic Sound Design’. In: Coëgenarts, M and Kravanja, P. (eds) Embodied Cognition and Cinema, Leuven University Press, pp.155-186
Adam Melvin is a composer and
lecturer in Popular and Contemporary Music at Ulster University,
Derry~Londonderry, Northern Ireland. A great deal of both his compositional and
research practice is concerned with interrogating the relationship between
music, sound, site and the visual arts, particularly moving image. He has
received numerous international performances and broadcasts of his music; his
research has been published in The Soundtrack, Short Film Studies (Intellect)
and in the Palgrave Handbook of Sound Design and Music in Screen Media. He is a
member of Dublin’s Spatial Music Collective.
Brian Bridges is a composer and
lecturer based in Derry~Londonderry, Northern Ireland, where he lectures in
music technology at Ulster University and is Research Director for music and
associated subjects. He is the current president of ISSTA (the Irish Sound,
Science and Technology Association) and serves on the editorial board of
Interference: a Journal of Audio Culture. Much of his work is inspired by
connections between perceptual processes, creative practices and technologies,
and his creative output includes sound–based installations, audiovisual pieces
and electroacoustic and acoustic composition, including microtonal and spatial
music. He is a member of the Spatial Music Collective and is represented by the
Contemporary Music Centre.