How to influence a person’s experience using sound ?
Sounds dynamically triggered by an interactive system can be used to draw a person’s attention towards specific events and processes, attune the person’s emotions and entrain the person’s body movements.
Sounds that accompany our everyday actions play a profound role in the way we perceive and interpret our activity and the world around us. A well-known demonstration of how what we hear serves as an interpretation frame for what we see is to change the soundtrack of a movie scene and thereby to modify the audience’s emotional evaluation of the very same scene (Kuleshov effect).
Sonic Interaction Design (SID, Franinović& Serafin 2013) deals with the design of interactive systems that interweave the user’s physical actions with sonic feedback and thereby affect the user’s perception, movements and emotions to incite intended experiences. Thus, SID can be understood as a means to design for experience by drawing on sound as a medium and the user’s listening abilities.
Challenging Interrelationships
Sound has beneficial characteristics to be used in interaction design for mobile activities. Supportive systems can use sonic representation to provide activity-related information preserving the users’ mobility and without demanding their full attention. However, designing sonic interactions also poses a number of challenges, such as understanding how the sounds that are elicited by the user’s movements affect the way the user perceives and experiences the activity.
At the same time that sound acts upon a person’s experience of an activity, so do a person’s activity and its contextual conditions influence the perception of sound. The particular context within which sound is perceived can be decisive for the sonic properties that the listener focuses on and hence color the perception and interpretation of sound.
Designing interactive sound differs from designing the soundtrack of a movie. The specific temporal arrangement of events is not predefined and the designer has to account for the dynamic interplay of various sounds. The designer needs to consider the overall soundscape that will evolve when different sounds are heard in succession, when they overlap, or when they are triggered simultaneously.
The experience of interactive sound in a mobile setting is affected by a variety of factors and substantially differs from experiencing sound in a static setting. This requires from sonic interaction designers a direct experiential awareness of the varying interplay of sound, body movement and use situations.
Sonic seduction builds on interactive sonic figures that evolve as a person’s movements gain stability. Sonic figures are used as means of negative feedback in order to disrupt routines.
Hajinejad, N., Grüter, B., Roque, L. (2017) Prototyping Sonic Interaction for Walking. In Proc. of MobileHCI ’17, New York, USA: ACM, pp. 98:1–98:8.
Karmen Franinović and Stefania Serafin, eds. 2013. Sonic interaction design. MIT Press, Cambridge, Mass.
