The Road to Dolby Digital 5.1 on The OG Xbox

Surround sound and immersive audio are so commonplace in gaming today -- to the point of almost being an afterthought -- it's easy to forget that the OG Xbox’s inclusion of interactive Dolby Digital 5.1 Surround was a groundbreaking innovation at the time, bringing literally new dimensions to how video games are experienced.

It’s hard to believe that 25 years ago, we were in the final stages of developing the original Xbox. As the Xbox Audio Program Manager, I was fortunate enough to be part of the team that helped shape the audio feature set for the console (and was also fortunate enough to have created the startup sound). One of the features we were most excited about was an industry first — real-time interactive digital surround sound, via Dolby Digital 5.1, built into the Xbox itself, available on every Xbox game.

Xbox's Dolby Digital 5.1 can be arguably traced back to a random lunch conversation at the Computer Games Developers Conference in Long Beach, California in 1998.

It came uncomfortably close to not happening...

The vision was easy to understand: You could experience full-fidelity game sounds from all around you, with pristine digital audio quality. Digital Surround sound showed gamers that the TV screen in front of you was just a small visual window into the game, but the entire world around you still existed. And now you could hear the rest of that world around you even if you couldn’t see it. Enemies behind you, explosions off to your right, reverberation coming from all around you. Although some earlier games supported an older, analog surround tech, they didn’t have the ability to deliver full-bandwidth, digital surround. And although digital 5.1 audio was technically on the PlayStation 2, it was limited to DVD movie playback or non-interactive cut-scenes, lacking the interactivity required for games. By designing it into the Xbox console itself, every game would be able to be experienced in 5.1, without using having to compromise other aspects of the game. Halo composer and audio director Marty O’Donnell has said that a big part of why he decided to go with Bungie and become a Microsoft employee was because of the promise that Xbox was going to have interactive digital surround sound.

Although the vision was straightforward, getting it to happen was far less so. There were two big challenges getting Dolby Digital 5.1 into the Xbox. The first was expected: the technological challenge. The other challenge was a complete blindside... How to get two big companies, each accustomed to having the upper hand in negotiations, to actually agree and sign a deal with each other….

What is Dolby Digital 5.1?

Dolby Digital (sometimes called “AC3”), was originally designed to put Digital Surround sound into movie theaters, providing 6 channels of audio around the audience, and was the first to define what "Surround sound" speakers should be: Front Right, Front Left, Surround Right, Surround Left, Center and “LFE” ("Low Frequency Effects" for the subwoofer). By 1997, it had become the mandatory audio standard for the newly developed “DVD” format for at-home movie playback.

Unfortunately, we couldn’t just take that technology as-is and put it into the Xbox. That would have been nice! To understand why, we first have to talk just a bit about Dolby Digital itself.

Dolby Digital solved two big problems when the DVD movie format was being developed. The designers knew they wanted a ‘theater experience’ at home, so surround sound was a ‘must have’ feature. Unfortunately, regular 5.1 DVD quality audio takes up a lot of space. If left uncompressed, just the audio for a 2-hour movie would be more than 6 Gigabytes; that’s more data than fit on the entire DVD, which was about 4.7 Gigabytes at the time. Dolby Digital solved this by compressing (encoding) the 6 channels of audio data and maintaining very high quality. By using the Dolby Digital algorithm, that same 2 hours of Dolby Digital 5.1 audio takes only about 350 Megabytes, leaving plenty of room for the video part of the movie.

The next problem was how to get the audio from the DVD player to the user’s home theater system. They could have designed the system such that every DVD player had 6 audio outputs, with receivers having 6 audio inputs… and a few did. But that was considered too unwieldy and not a very elegant solution. So, they decided instead on a format called S/PDIF, a technology co-developed by Sony (the “S”) and Philips (the “P”) to transmit high quality 5.1 digital audio with just a single cable. At the time, the main way was by using a TOSLINK optical cable, or by a Digital Coaxial cable.

S/PDIF Digital outputs from a DVD player

That was the birth of the home-theater audio ecosystem. DVD’s would ship with the Dolby Digital compressed/encoded audio on the DVD, and those wanting surround sound would connect their DVD to their home theater receiver by S/PDIF cable. The receiver would then decode the compressed audio back into 5.1 discrete channels, which would play through the viewer's speakers.

For movies, that works great. For games, it's doesn't work at all.

The problem is a fundamental difference between how games and movies are made and experienced. When a DVD movie is created, the entire experience is created at the movie studio, during production. For sound, the studio creates the 5.1 soundtrack in a sound studio and processes it with Dolby Digital Encoding software. The software takes the 5.1 channel movie sound mix created at the studio, and compresses it into a Dolby Digital format, which is then put on to the disk along with the video data.  That’s the key point here. The 5.1 audio for a movie is encoded at the DVD’s production time; then the data is decompressed in real-time by the home theater receiver and sent to the 5.1 speakers as the movie plays back.

Movie 5.1 Surround Mixing (Production Studio) and Playback (Home)

Movie Sound is mixed to Dolby Digital 5.1 at the Movie Production Studio (left). Movie Sound is decoded from Dolby Digital 5.1 and sent to speakers in the viewer's living room (right). Pay attention to where the "Dolby Digital Encoder" is.

By the year 2000, DVD’s and home theaters had been around long enough that Dolby Digital, mastered into the movie at production time, sent from DVD player to receiver via S/PDIF and then decoded by the AV receiver during playback had become the de-facto way of delivering surround sound in a home theater system.

New Xbox Challenges: Interactive Digital Surround Sound

When we were designing the audio system for the Xbox, we knew we wanted to include full-fidelity, digital surround sound. We realized that we were never going to get masses of people throwing out their existing AV receivers and re-wiring their home theater systems with some new “Xbox” surround format. So, we had to find a way for the Xbox to use people’s existing home theater setups. Receivers of that era all supported Dolby Digital Surround and many also DTS (a Dolby competitor) Surround Sound. We won’t get into the reasons we went with one over the other except that unless we wanted users to have to replace their home theater systems, we knew we had to go with one of those formats.

The big challenge was that Dolby encoding was never designed to operate in ‘real-time.’ As we mentioned, for a movie, that’s a process done as part of the DVD production process. For a game, that doesn’t work.

That’s because when you are playing a game, the experience is generated moment by moment inside the game console itself. You don’t know how a sound (let’s say a helicopter) needs to be placed around the player until they are in the middle of playing the game. So you can’t encode the game’s audio into 5.1 ahead of time like you do for a movie; if the player just turns their head to the left, the sound of the helicopter needs to move to the correct speakers, reflecting the new orientation of the players head to where the helicopter is in the game.

That means that any surround sound encoding needs to happen in real-time, as the player is playing the game. Which means that the Dolby Digital encoder, which takes 5.1 audio created by the game and changes it into compressed data to be output to the S/PDIF output, needs to run inside the Xbox, in real-time as the game is played.

Game Sound Creation (Game Developer) and 5.1 Mixing Inside Xbox and Playback (Home)

Game Sound is put into the game as individual sound effects by the Game Developer (left). Game Sound effects are mixed and encoded to Dolby Digital 5.1 in the Xbox and sent to speakers in the viewers living room (right).

Note that the Dolby Digital 5.1 encoder is inside the Xbox itself

My first conversation about real-time surround encoding for games occurred well before the Xbox had even been conceived. In 1998 I’d very recently been hired by Microsoft to head up sound for DirectX, the technologies Microsoft created to make it easier for developers to create and support games on Windows. One of the things I was tasked with was thinking about how we (Microsoft) could get Windows into people’s living rooms. Yes, that was a thing…

For me, from a sound perspective, that meant “we need to take advantage of the home theater in players’ living rooms and support surround sound for Windows games.” For all the reasons mentioned earlier, I was very interested in seeing if Windows could create encoded 5.1 surround for games. I recall having a lengthy lunch with a Dolby engineer where everything came down to latency.

The Latency Problem

Latency is the time from when we tell a sound to play, and it actually coming out of the speakers. If the latency is too long, the sound for the game will feel out of sync, like a badly dubbed movie.

I was having lunch with an engineer from Dolby during the 1998 Computer Game Developers Conference in Long Beach, CA, and the question of latency came up. We were doing some ‘napkin sketch’ ideas on how it might be possible to do digital surround for interactive games in Windows. He said that, unfortunately, Dolby Digital probably wouldn’t work. the latency of encoding Dolby Digital 5.1 was too high; if I recall, he said it was over 200 milliseconds. At that large a latency, you’d see an explosion but have a very noticeable delay before you heard it. He couldn’t see any way it could get down to below 10 milliseconds, which he assumed was the target. Less than 10 milliseconds is the kind of latency you want if you’re making a musical keyboard, for example.

But for games, it turns out, are a bit more tolerant of longer latency.

I knew that the existing Windows game audio system had much higher latency than 10ms, having spent some time prior to Microsoft analyzing DirectSound’s performance and latency. Yes, lower is ideal, but games can still feel pretty ‘crisp’ and in sync with the visuals even if the sound latency is as much as 50, maybe even 60 milliseconds. I mentioned that 50 or so milliseconds might be a usable target.

As an aside, latency is also the reason that music games, such as Guitar Hero or Rock Band, are among the very few Xbox games that disabled Dolby Digital Surround by default. In those games, you really do need the audio latency to be as low as it possibly can be for the games to ‘feel’ musical.

That was pretty much the end of the conversation, until about a year or so later. I’d moved over to the fledgling Xbox group, which had become a real, if not quite fully green-lit, project, at Microsoft, and it was time to talk about real-time Digital Surround Sound again, this time with some urgency. We picked up from where we left off at the restaurant in Long Beach, and got serious about trying to make it work.

First, we verified that the latency targets of 50-60 milliseconds would really be ok. Prior to joining Microsoft, I’d done some latency measurements of Windows gaming audio system, and it was generally around 30-50ms. So clearly, it was possible to have a great game audio experience with latencies that high. To double check, I played a bunch of original PlayStation, Sega Genesis and SNES games, and ran their output through a digital delay, cranking up the delay until the games felt obviously out of synch. And sure enough, it wasn’t until it got longer than that 60-70ms target before things felt a bit off. I wish I could say we did a thorough, double-blind test with dozens of subjects to verify this… but we didn’t. Xbox moved pretty fast in those days, and that was pretty much the extent of our 'testing!'

Armed with that latency data, and due to a bit of luck (some would say ‘good engineering design’) and some clever programming, all the pieces were about to fall into place.

The first bit of luck was that the Dolby Digital 5.1 standard that had been shipping since the first home theater receivers were created actually supported more than one Dolby Digital Surround format. In practice, only one was typically used on movie DVDs. But the other 2 formats were part of the official Dolby Digital specification, so every receiver had to support them. And it turns out that one of those formats looked like it might allow us to meet our stringent latency requirement.

And it turned out it could, thanks to an incredibly clever modification by Dolby that tweaked how their Dolby encoder worked. A good analogy for what they ended up doing might be if you called an Uber to bring you to the airport, but you hadn’t even started packing yet. But you know that as long as you can get packed before the Uber arrives at your house and starts driving away, there’ s no problem. In fact, you time it so that just when the Uber is just about to give up on you and leave, you get in. You can read what they did in this patent

With that invention, real-time interactive Dolby Digital became possible. Between that bit of good fortune, and the clever engineering, we knew it was theoretically possible to encode a game’s audio into Dolby Digital 5.1 with low enough latency to have an excellent game experience!

From a system design perspective, that meant we now had to make sure there was enough processing power in the Xbox audio system to do the encoding without affecting other aspects of the game. By this time, we were narrowing our audio solution for the Xbox to what would eventually be called the MCPx chip, manufactured by nVidia. The Xbox audio chip was incredibly powerful and was based on the “Recording Studio on a chip” architecture that we had been rolling out in DirectX 8 on Windows, including a programmable DSP (Digital Signal Processing) chip for audio effects processing. After some detailed discussions with nVidia and Dolby, we decided to add an entire second programmable DSP dedicated to the new low-latency Dolby algorithm.

A huge challenge was that there was no way to emulate this. We literally couldn’t do the work until we had actual prototype chips on an actual prototype Xbox circuit board. I don’t recall the exact date those finally arrived, but it was only a few months before the Nov 15, 2001 launch. Not a lot of time to verify a brand new algorithm on a prototype DSP plugged into prototype “alpha” hardware. Fortunately we had some wicked good programmers on the Xbox audio team. They wrote high-level C and low-level 56000 assembly DSP code, somehow taking an algorithm that required 24k of ram and making it work with only 6k, enabling the sound chip to generate complex game sounds, perform studio-quality audio processing, and output a Dolby Digital 5.1 Surround Sound signal ready sent to the gamer’s home theater system, plus invent and write new testing methods and code to make sure everything was working properly-- all in just the few months between receiving the prototype hardware and needing to provide code to game developers in time for them to ship their games by launch.

That left the business side.

For obvious reasons, I can’t really go deep here, except to note a couple of things. At the time, the two companies, Dolby and Microsoft, were each used to being in the driver’s seat when it came to negotiating licenses and contracts. If you wanted to make a DVD player, you had to get a license from Dolby. There was no choice. Likewise, if you wanted to ship a PC, you had to license Windows from Microsoft. There was no choice.

Let’s just say that getting two companies, each used to being in a ….very strong.... negotiating position to come to an agreement was challenging; the biz dev and legal team had their hands full. It took the better part of a year to get the deal done and things came close to falling apart more than once.

It all came down to one evening. We got a call from a production Manager, who was at the factory in Guadalajara, Mexico. That next morning, they were scheduled to start pre-production of the first prototype Xboxes. They essentially said: “we’ve got two molds for the Xbox outer casing ready to go. One has the Dolby logo on it, one doesn’t. We need to know which one we should use…by tomorrow morning.”

Xbox with Dolby Digital and Microsoft Logos molded into Xbox casing

That evening, over late night phone calls, the final agreement to include dynamically encoded Dolby Digital with every Xbox was made, and we gave the manager the go-ahead to use the mold with the Dolby logo. The next morning, they FedExed the contract to us for signing. By coincidence, the day the contract arrived, there was an Xbox all-hands meeting (probably to celebrate the first Xboxes coming off the assembly line). The contract was literally signed at the all-hands, in front of the entire Xbox team not even 7 months before launch. 18 months of investigation, engineering, business and legal issues paid off, and every Xbox game would support interactive Dolby Digital 5.1 Surround Sound.

When first announced in April 2001, it was somewhat clunkily called the “Dolby Interactive Content Encoder.” They latter dubbed it “Dolby Digital Live.” (The new name was a little weird for us at Microsoft, since we knew we would be shipping “Xbox Live” for online gameplay in about a year, but that’s what it was).

A month later, we announced the launch date. From there, it was a race to Xbox launch, with the ATG (Advanced Technology Group) ensuring that Xbox game developers understood the new system, and just, exactly what it meant to mix games in surround sound, how to use the LFE/subwoofer channel most effectively, how the tools and API’s worked, and making it as straightforward to use as possible, with launch title, HALO, being a hard-core user of DD 5.1. The ATG team logged a lot of road miles, and I recall two literal round-the-world flights as we talked to developers in London and Tokyo about the Xbox audio system leading up to the November 15, 2001 launch.

Being a part of getting real-time Digital Surround into gaming, an industry first, was incredibly rewarding, watching multiple engineers from 3 companies doing some cutting edge engineering and tough programming. It was particularly fun having audio being called out in marketing materials, including the front of the Xbox packaging itself, and seeing various peripherals

like “Xbox Dolby Digital Surround Speakers” and “Xbox Dolby Digital Surround Headphones” appear on the market.

One particularly memorable moment came shortly after we had just barely gotten things starting to work. A few months before launch, Bill Gates spent a full day at Xbox, looking at everything; hardware, graphics, the controllers, as well as audio. We had a primitive demo

program that let you move a helicopter sound around you in 360 degrees, using the controller. It was buggy.. the helicopter sound would frequently pop in and out. But as you moved the joystick on the Xbox controller, the sound was clearly circling around you, coming from the rear speakers as necessary, filling the room with sound. At the end of the day, the entire Xbox team met in the cafeteria, and Bill gave his thoughts on his day at Xbox. Someone asked the question, “What was the most interesting thing you saw today?”  He stroked his chin for a few moments and said, “I think it might be the audio stuff.”

Today, of course, surround sound in games is a given. Dolby Atmos and VR technologies provide a full 360 sphere of sound; front, back, up, down. The PS5 has dedicated hardware for virtual surround sound for 5000 sound sources.. and it’s just…. there.

But when the Xbox launched back in 2001, the concept of multi-speaker digital surround sound in games wasn’t a thing, and full-fidelity digital surround sound was brand new with the Xbox.

Brian Schmidt was Program Manager for Xbox Audio from 1999 - 2008 and has been creating sound and music for games since 1987.

Read more at the GameSoundCon blog