Compressed Audio vs. CDs: Can You Tell the Difference?
Are you able to distinguish between the two? Our tests reveal some surprising results for those who love--or hate--listening to MP3s.Richard Baguley
Many people complain that compressed audio doesn't sound as good as CD audio. To verify that claim, we subjected a group of typical listeners to blind tests to see whether they could tell the difference between music from CDs and compressed versions of the songs in a variety of file formats. We also asked the testers to rate the compressed versions on a quality scale.
Before you read the results of our tests, bear one thing in mind: The quality of compressed music is very much dependent on how you play it back. A compressed file will sound considerably different when played through a high-end sound card and headphones (such as those we used in our tests) than it will through a portable MP3 player with a pair of cheap earpieces. We did not test any portable players; our primary goal was to test the quality of the compression formats, rather than the quality of hardware.
How We Tested
From each of five pieces of music, we ripped a 30-second clip from a CD to a.wav file. The five music selections were an acoustic version of Daughter by Pearl Jam, Radioactivity by Kraftwerk, a cellos-only version of Wherever I May Roam by Apocalyptica, a live version of Time by Pink Floyd, and O Fortuna from Carl Orff's Carmina Burana. These presented a range of challenges to the compression programs, as they ranged from subtle acoustic sounds to full-on orchestral splendor.
We compressed each clip to a variety of bit rates using Windows Media Encoder for the WMA format, Real Jukebox Plus for the RealAudio format, MusicMatch Jukebox Plus for the MP3 format, and RealJukebox Plus with the Dolby AAC plug-in for Dolby's AAC format.
We then played back both the original files and the uncompressed versions to 30 testers (mainly PC World staff, but we also got representatives from Dolby laboratories, Microsoft, and Real Networks to participate). We asked them to identify the compressed versions. We used a double-blind testing methodology in which each tester was given three files to listen to, called A, B, and C: The A file was the.Wav file ripped directly from the CD. The other two were either the.Wav file or a compressed version.
We then asked the testers to rate what they thought was the compressed file on a quality scale of 1 to 5:
- 5- No perceptible differences between compressed and uncompressed
- 4- Perceptible differences, but not annoying
- 3- Slightly annoying differences
- 2- Annoying differences
- 1- Very annoying differences
The files were played back on a Dell system with a Pentium III-600 CPU, 128MB of RAM, and a CreativeLabs Sound Blaster Live Platinum sound card and its optional Live Drive component using Cakewalk Sonar, which allowed the testers to listen to the files side by side and to switch between them at will. The testers listened to the sound clips through Sony MDR-7506 headphones connected to the Live Drive headphone socket.
Test Results (chart)
| Codec | Bit rate (kilobits per second): 64 | Bit rate (kilobits per second): 96 | Bit rate (kilobits per second): 128 | Bit rate (kilobits per second): 192 | Bit rate (kilobits per second): 256 |
| AAC | 12% | 52% | 70% | N/A | N/A |
| Real1 | 36% | N/A | 69% | 81% | 79% |
| MP3 | 5% | N/A | 69% | 70% | 73% |
| WMA | 9% | 11% | 14% | N/A | N/A |
Test Results Explained
The first set of results shows the percentage of testers who felt that they couldn't tell the difference between the compressed and uncompressed versions. We were not surprised that we found the higher the bit rate, the higher the percentage of testers who couldn't distinguish between the files. However, that's not the whole story.
The codec that came off worst in the tests was MP3. At the lowest bit rate tested (64 kbps) only 5 percent of the testers could not tell the difference in quality. The live Pink Floyd and acoustic Pearl Jam tracks were particularly easy to distinguish: Both pieces contained a variety of subtle sounds (such as audience noise in the background of the Pink Floyd live track) that were lost or garbled in the compressed version. Every one of the testers could tell the difference between the compressed and uncompressed versions of both tracks.
The other interesting note is that the Real codec did the best at the lowest bit rate: Just over a third of the testers couldn't tell the difference between the uncompressed version and the version compressed at 64 kbps.
How the Formats Compare
| CODEC | Tested bit rate (kilobits per second): 64 | Tested bit rate (kilobits per second): 96 | Tested bit rate (kilobits per second): 128 | Tested bit rate (kilobits per second): 192 | Tested bit rate (kilobits per second): 256 |
| AAC | 3.4 | 4.6 | 4.8 | N/A | N/A |
| MP3 | 2.2 | N/A | 4.7 | 4.8 | 4.9 |
| RealAudio1 | 4.1 | N/A | 4.8 | 4.9 | 4.8 |
| WMA | 3.6 | 4.1 | 4.1 | N/A | N/A |
Understanding the Formats Comparison
When our testers rated what they guessed were compressed tracks, all of the formats scored above four at a bit rate of 128 kbps or higher. We were surprised that MP3 came out on top with a score of 4.9 at the highest bit rate of 256 kbps. However, RealAudio encoded at 64 kbps also scored an average of 4.1, significantly better than the other formats at that low bit rate.
Interestingly, our testers did not rate WMA-encoded files at 128kbps any higher then WMA files at 96 kbps, and at 128 kbps that format was rated significantly lower than the other formats--even MP3.
Although MP3 is the most widespread format, it performed the worst at 64 kbps, achieving an average score of only 2.2.
Judging from our tests, we think most people will find music compressed at higher bit rates indistinguishable from the original versions. Music fans will probably find even moderately compressed songs acceptable, especially if the compression means they can fit more tracks onto a portable player. However, this compromise becomes less tolerable when it involves compressing the music to the lowest bitrate.
For general use, 128 kbps is fine (or 192 kbps if you have really good ears). If you want to pack a lot of audio into a small amount of memory space, you can take the bit rate down to 64 kbps for RealAudio and 96 kbps for AAC and WMA. Tthe lowest rate that is really usable for MP3 is 128 kbps; any lower, and the quality begins to suffer significantly.