Before the birth of the web, most developers knew nothing about visual design and had never worked with a graphic designer. Now no application is created without one. In game development, every team works with sound designers and composers to create the sound effects and music for their games. So why have most non-game software developers never used audio in their applications? Three game sound designers explain how sound can make your app better.
From the beeps and bleeps of early PC games to the cinematic soundscapes of triple-A titles, audio has always been crucial in games. Sound designers determine the audio direction (as the art director does for visuals) of a game and then record and manipulate appropriate sounds. Composers create music.
Audio is used in games to communicate information and to heighten emotion. In simpler or more casual games, information flow is often the primary focus. “A typical example is in the very first Mario games, where the music actually sped up when you had little time left,” says Mattias Häggström Gerdt, who composes the music for Mojang's game title Scrolls.
In applications, audio information flow usually means gestures and notifications. You might have UI sounds associated with usage of buttons or sliders, while other sounds indicate notifications or give error and success messages. “Music also does a lot to set the pace of the gameplay,” adds Gerdt. “Scrolls has a slightly slower pace so we try to go a bit more wide and use silence more than you would think to let the music breath a bit. It's very much a strategy game so you need time to think about your next move.”
Emotion is the flip side of game sound design. David Mollerstedt headed up the audio group at EA's DICE studio for the Battlefield and Mirror's Edge titles. “The games we did at DICE were building on emotions,” says Mollerstedt. “Sound is kind of a secondary sense in terms of visual being the primary sense, but the flip side of that is that it can go directly to the emotions. It's much easier to build strong emotions with audio than it is with video. Video is very strong and direct but the audio is very subconsciously strong, so when you can align that you get something which feels very cohesive.”
Audio can make your app more useful, more enjoyable or more addictive. You can use sound for branding, to give information or rewards or to heighten emotion. But the first issue to consider is attention. Most games are designed to be immersive; they expect you to focus on them completely.
When integrating audio, you need to consider how much attention users will devote to your application. Will they run other software in parallel? In this case a full soundscape won't be suitable. Sometimes silence is the best option. All three sound designers emphasised that the purpose of audio is to enhance the game or application, not to distract from it. “It's not about doing good audio. It's about doing a good game,” says Mollerstedt.
Associating a sound effect or music with your visual logo is one the simplest and most effective ways to use audio in your application. “It's really iconic the way a Mac sounds when you start it up,” says Gerdt. "Having those sound logos can mean so much when remembering a piece of software or a brand. One of the strengths of the first video game music that got really famous--the classic Super Mario themes--was that everyone knew that was the Super Mario theme.
The composer of that music is a big Beatles fan and the highlight of his career was when he heard one member of the Beatles whistle the Super Mario tune.” Sound logos can be used when your application starts up or can be integrated into video content like tutorials associated with your application.
Josh Mobley is a composer who also does sound design for iPhone applications. “I've often argued that notification sounds are your branding sounds. If you get a notification from Facebook, you know what that sound is. The developers of popular applications have the opportunity to make these iconic sounds which can let other people know, for good or ill, what applications you are using. It's a way to communicate without any kind of visual presentation.”
Combining visual and audio feedback helps the user to identify actions and results in your application. Notification sounds can bring the user's attention to something even when there is no visual information. When you can hear that you have clicked on something, it also gives more complete tactile feedback than if you just see it.
Ideally all these forms of audio feedback should be designed in a coherent way in parallel with the visuals. Mobley explains how this works in To Do list application Clear. “Clear is unique. We wanted it to be a musical instrument. Everything's in the same key and when you complete a task it's supposed to feel like Valhalla almost, an epic task. Moving up and down the hierarchy of menus invokes string sounds. The smaller gestures are really small sounds which are not musical, to not distract.”
Sounds are often used in games as a reward when you complete a level or succeed in some task. In a similar way you can use audio rewards in your application to encourage users to perform tasks, they might not find very enjoyable. “The experience needs to be half work and one part entertainment and that's where sound comes in,” says Mobley. In Clear if you complete three tasks quickly the sound becomes higher and higher. Once you clear the entire list of to dos, you get a little jingle. “In Clear the idea was to make it almost Pavlovian,” says Mobley. “You wanted to complete that task because you wanted to hear that sound.”
There are plenty of stock sound libraries available, and these can suffice for some applications, but if you are serious about sonic branding you need to hire a sound designer or composer to create original sound effects and music. ”This shit costs money,” says Mobley. “It can literally range from $300 at the low low low end to five grand depending on the scope of the project.”
All the designers told me how important it is that developers learn to communicate with the sound designer. ”We have a quite advanced language for visuals,” says Mollerstedt, ”but if you try to talk about audio, people have very different opinions about what a word means. The language is not that developed. It's the same with smell. If you want to work with audio it's really important to agree what things mean and build a language.”
Some developers have very specific ideas about the audio. Others will look to the designer to decide on the feel of the sound. Mobley says that there are sometimes disputes about length and musicality. “Developers always want the sounds to be shorter because they want to make their app smaller. I tend to like sounds that have some texture to them, some meat.”
The process will vary from designer to designer. Mobley asks developers to provide a video of all the application's user functions.“I need the timing. If there's any kind of animation for panel sliding or whatever I want to get that exactly right. Then they don't have to inject the sounds into the app and recompile, they can just look at the video and see what's working and what isn't working.”
Several iterations may be needed to achieve cohesive sound and visuals before the audio is actually integrated into the app. But the results are often worth it. “People say that sound is half a picture in film,” says Mobley. “I would argue that that's also true in games and apps. If you are able to make that reward sound just right, it keeps them coming back.”
[Image: Flickr user Vancouver Film School]