Engine Feature - Audio Subsystem

This is a proposal write-up for my upcoming 3-week assignment. As a part of our final project, we have to create our own engine sub-system which can be integrated in our existing system. Few of the things which should be kept in mind while designing this sub-system are:

• It should be easy to understand and implement for anyone who wishes to use this system.
• Platform dependencies should be efficiently handled. This means we should have platform independent interfaces and platform dependent implementations.
• Like our graphics system, we should have a builder which should build our human readable files.

For this purpose, I have decided to implement Audio Engine Sub-system in our engine. I felt it would be interesting to work with audio on a low engine level. I am not sure how much I will be able to implement in three weeks, but below are some of the feature of I would like to implement:

• Load an audio file, possibly support various audio formats.
• Play, pause and stop an audio.
• Rewind or forward an audio.
• Trim an audio file: ability to specify start and end time for an audio.
• Loop an audio.
• Adjust the speed of play: like 2x or 0.5x
• Ability to adjust the audio properties like pitch, treble etc. for a file (though I believe it is a stretch goal at this point)

It’s an uncharted territory for me and I am excited to deep dive into this engine feature. I believe it is a great learning opportunity for us as well as a great place to implement what we have learned throughout the semester. More than implementing additional features, my priority will be on the architecture on this engine sub-system.

Update - 11/08/2018

xAudio2
After doing some research of various audio APIs, I have decided to go with xAudio2 for my project. xAudio2 is a flexible, cross-platform, low level audio API which came as a replacement for DirectSound for windows applications. The three main components in xAudio are:

• Source Voice: We create a source voice when we want to play a sound. We can set various parameters on voice such as pitch, volume etc. Once we create a source voice, we point it to a piece of sound data in memory and play it.
• Submix Voice: It is similar to source voice but with few differences. Firstly, the sound data of submix voice is not a piece of sound data in memory, but rather the output of another source or submix voice. Secondly, a submix voice can have multiple inputs. These are very useful in creating complex sound effects from multiple wave files.
• Mastering Voice: There is only one mastering voice and its job is to create the final n-channel (2.1, 5.1 etc.) output for speakers. It takes input from all source and submix voices and combine them to present one output.

With the above components in mind, the high-level workflow of xAudio2 is as follows:

• We first create the xAudio system using XAudio2Create.
• Then we create our mastering and source voices. These voices are used for processing the audio data.
• Then we load the audio data for source voice. We generally use a buffer (which can be useful if we try to implement the streaming audio) for loading the audio data and pointer to this buffer is passed to the source voice.
• Source voice then process this data and pass in onto the mastering voice for the output.
• Streaming Audio: Multiple buffers can be submitted to the source voice so that it can seamlessly process the various buffers. This way the audio is processed in chunks. The actual processing of buffers can be done on a separate thread. This is known as streaming audio where we utilize multi-threading to load and process the chunks of sound data. xAudio provide us with various callback functions which we can override to signal certain events (like buffer end or stream end) while streaming audio (more on this when and if I actually implement this. My currently priority is just to play a sound, streaming or not!).

Wave File Format
In my audio engine, I am going to use the wave files (maybe I will be able to support the multiple formats, but first goal is to play a single format correctly). It is important to understand how the data is actually stored in wave files in order to play it correctly. Below are some details of a wave binary file:

• The first 12 bytes are the RIFF chunk descriptor which store the following information:
  • First 4 bytes: chunk ID
  • Next 4 bytes: length of the file
  • Next 4 bytes: format i.e. wave
• The wave format requires two sub chunks; fmt and data. The fmt chunk describes the sound sample format such as sample rate, number of channels etc. the data chunk contains the audio sample data to be played.
• The first 12 bytes of any wave file is fixed. Usually we get the fmt chunk after these 12 bytes but it is not always certain. It is generally advisable to perform some sort of check before reading the chunk as ‘fmt’ or ‘data’. We can read the first 4 byte for ‘fmt’ or ‘data’ to determine this. I found the below image which describes the wave file format pretty well.

WAVE File Format

Update - 11/15/2018

I was able to play a wav file with my implementation this week. Though there are some errors, mostly in cleaning up of the engine system (destructors), the sound is playing fine. Below are some of the learnings and implementations I made this week:

• xAudio.lib is not platform independent and it is important is to realize for which windows we are building our application and include the appropriate xAudio files and library. xAudio comes with the DirectX SDK in previous windows before windows 8 and in WindowsKit folder in and after windows 8.1. We can specify our application to specifically target a version of windows, but it was not advisable in my case (as no other student is using windows 7 or below and we have to make the system for everyone to use). I didn’t use the DirectX SDK but rather utilized the xAudio which was present in my version of windows.
• In windows kit, we can find the xAudio library file for both the x86 and x64 architecture. I am using preprocessors to define these pre-built libraries. The name of the files is not what I have shown below, but it is xAudio.lib for both the architectures. I have named them according to the configuration for better readability.
• [Edit: We don’t need to do this, windows will automatically detect the configuration and use the appropriate library.]

• For playing a sound, I have created 3 classes:
  • Sound: Class that loads the wave file from a file path
  • Sound System: Audio Engine; there will only be one instance of this in our game and it is used to play the sound. We do XAudio2Create and create mastering voices in the constructor of this class.
  • Channel: This has the source voice pointer and xAudio2 buffer. Any callbacks to be overridden will be done here. This submits the source buffer data and start the source voices.
• Reading of wave file is one of the most important things while playing any audio file. As explained in last week’s update, I am using the similar approach to read the wave file. But my system is not that robust at this point and some of the wave file are throwing errors.
  • Different wave file can have different format properties (like block align, number of channels, basically anything we read in ‘fmt’ portion of the file). It is important to make a system robust enough so as to deal with multiple type of format properties. Currently I am setting my default values for one system so its failing for some other type of format properties. My priority will be to complete the implementation of other feature of the system and then deal with different formats of the wave file. I am using these default values in my SoundSystem initialization: 

• To explain the audio system workflow; we read the wave file in sound class. Sound Engine sets the default format properties for our wave file and create a xAudio object and a mastering voice. Loaded sound is stored in xAudio buffer as source voice and buffer is submitted to play the it. Channel class is only used by the sound system class, we can make it internal to the SoundSystem. I might implement this based on the complexity of the system.
• Below are the interfaces I am using for each of the three classes:

I am expecting to update these interfaces in the coming week. As for the last week, I want to achieve at least the following:

• Fix all error from the system and improve its interface.
• Add this project to our graphics project.
• Pause and Stop Audio, adjust playback speed, volume.
• Support multiple wave formats

I am hoping I will be able to achieve these goals (and maybe some stretch goals). Though being annoyed by not being able to make thing works half of the time, I am really enjoying this assignment. The system is more complex than I expected, but I am learning a lot from this. Really excited (and somewhat scared!) going into the final week.

Update - 11/21/2018

Final goals achieved in this project:

• Able to play the sound without any errors or memory leaks.
• Public interface is platform dependent now. Users just need to use the Sound class for playing the sound.
• Integrated the Audio engine subsystem with our current engine. Added an audio builder application for copying the wave files from the game content folder to our game install directory.
  • Audio builder’s only job is to copy the wave file. We could also use the custom build step in any of our projects to copy the files. But in that case each user will be required to update the project settings with their files; and a copy call will be required for each file we want to move to the game install directory.
• Added functions for play, pause, stop, increase/decrease playback speed and volume.
• Ability to play multiple sound (some sound effects and the background music together)
• All types of wave files can be played using this now. I have removed the asserts checking against the wave format properties which we were setting in the sound system. It makes the system more prone to errors as corrupted wave file with invalid properties won't be detected, but it allows us to play all formats.

Sound Class
To play a sound using this project, other students will only have to use the sound class. It is important to remember to include the xAudio.lib in our project before using the sound class. Also, I am using the AudioBuilder project to copy the sound file from content folder to game install directory folder (as we have done before by using AssetsToBuild.lua file). The final interface of the sound for this class looks something like this:

This is how we use the sound class. We just need to call the sound functions on the object

Possible changes that could be made on this interface are:

• Instead of separate PlayInLoop() function we can have a bool to set if we want to play in loop or not.
• We could even allow passing the float value for increasing/decreasing volume and use that to directly set the value. I am currently changing the volume by 0.1f if we increase/decrease the volume.
• The sound is not streaming. It is loaded fully into the memory and then played. For streaming sound, we could use the buffers to read audio data in chunks and the xAudio callback functions to notify when the buffer is complete. My knowledge of streaming audio is still not complete at this point and I would like to integrate it on my system before semester end.
• When we increase/decrease sound or playback speed, its local to that sound file. It will not change for any other sound files playing in the system. We could have an interface to change these settings globally (i.e. change sound system).

Playing Multiple Sounds
Channels are used to play the sound. In my sound system class, I am maintaining two lists of active channels (channels which have sounds to play) and idle channels. Using these channels, we can play the multiple sounds (by making more channels active form idle). I have set the limit on number of channels to 64 (I don’t think we need more than 64) in this project.
[Update 1: Added an external.windows.h file to link xAudio2.lib. Now other student won't have to write any code to use the engine. This file will be forced include to the engine library.]
[Update 2: Added an interface for setting the volume and playback speed in the sound class.]
Downloads
Download Audio Project
Download Audio Builder Application
Demo Exe
A looping background music is playing. 'Space' to listen to a sound effect. '+' or '-' to increase or decrease the volume.