Signals and Noise
When I looked at those numbers and comments, I found that, for this jury at least, the type of compression mattered more than the level of compression, and that neither mattered all that much.
Basically, the scores were all good. The lowest score anyone gave anything was 3, defined in the jurors' instructions as "Definitely inferior, but you could still enjoy the music." Most scores were 4 ("Slightly inferior, but not enough to affect your listening pleasure") or above.
But Microsoft's .wma compression proved slightly superior. Compressed to the same level (128kbps), .wma files received generally higher scores on average than either .m4as or .mp3s--and that held true on both clips.
In fact, jurist Fran Avni (a singer, songwriter, performer, and music producer with more than 30 years experience) felt that .wma compression actually made both clips sound better than the uncompressed orginals. While she didn't agree, Madeline Prager pointed out that the compression masked musician errors. Madeline has played First Viola in several orchestras, and now teaches at the San Francisco Conservatory of Music. (Full disclosure: She's also my wife.)
Not that .wma's advantages were overwhelming. The format's scores averaged 4.51. The .m4a files averaged 4.26; the .mp3s, 4.01.
No clear pattern appeared when the jurors compared compression levels. I wasn't surprised that 192kbps .mp3 compression scored better than the same format at 128kbps. But I was taken aback that its average score of 4.19 beat out the scores for 256kbps and 320kbps .mp3s, and even the uncompressed .wav file.
The judges particularly liked the192kbps .mp3 version of the Tchaikovsky clip, which earned an average rating of 4.28 (still lower than the 128kbps .wma and .m4a clips). Steve Carter, a pianist and recording engineer who's played with Pete Escovedo, Ray Obiedo, and Taj Mahal, described it as "pretty close, maybe less smooth" than the uncompressed original. Steve rated it 4.8--the highest score he gave anything.
But the Joan Osborne clip compressed to192kbps .mp3 didn't do anywhere near as well with the judges. Its average score of 4.10 still did better than the 256kbps version's 3.63, but here the 320kbps and uncompressed versions beat it out--as you would expect--with 4.50 average scores.
The big surprise was that the uncompressed .wav versions of the two songs didn't get the highest scores.
The Tchaikovsky clip got an average rating of 3.80. The uncompressed Osborne version did better, with an average score of 4.5. But only one judge, Dan Howard, a musician, singer, and music teacher who has worked as a radio DJ and concert producer, correctly judged it "As good as [the] original."
How did so many trained ears fail to recognize the higher quality of less compressed tracks? While the result may seem like a fluke, it's not out of line with other, larger studies of music compression.
In 2001, the PC World Test Center conducted extensive audio compression tests, using four music samples, 30 judges, and a more controlled audio environment. While that study didn't test whether jurors could distinguish an uncompressed clip, it did find that listeners could detect very little difference between a clip at 128 kbps and one at 256 kbps.
And tests by Stanford University Music Professor Jonathan Berger indicate that young people are actually coming to prefer the sound of compressed music. Berger sees an increasing preference for MP3s and believes that the students he tested like the "sizzle" or metallic sound that the format imparts.
The bottom line from my testing seems to be this: My jury of professionals generally preferred WMA compression, but weren't able to detect much difference in files with bitrates higher than 192 kbps. In other words, even though they call it lossy compression, you may not be losing as much as you think.