Propeller Head Plaza Technical and scientific discussion of amps, cables and other topics.

---

[ Follow Ups ] Thread: [ Display  All  Email ] [ Propeller Head Plaza ]

---

[ Alert Moderator ]

---

"Objective" audio tests are not objective: An inside view on where objective audiophiles go wrong with their blind tests

Posted by Richard BassNut Greene on January 9, 2007 at 09:39:52

First, there is no debate that blind test results can only apply to the people involved, the stereo used, the songs listened to, and even then, ONLY on the day the test was performed. There is no reason to assume the same people would have the same results the next day -- in fact the results would almost certainly be at least slightly different the next day.

Here are many of the ways blind tests can fail to replicate the obviously superior subjectivist audition methodology (long-term listening at home):

For blind tests the stereo and room are unfamiliar to all but one listener (the homeowner).

The sighted warm-up audition will include A-B volume-matching and an A/B switchbox in the circuit -- A-B volume-matching is not done at home when listening to music, and there is no A/B switch in the circuit at home.

Some of the people will usually claim to hear differences in the sighted warm-up audition before the blind test, assuming there is one.

Those who don't hear differences, or are not sure, should be rejected from the upcoming blind test, but rarely are.

They could skew the overall results because they have no obvious sound quality difference claims to be tested, and could quickly become bored and not listen carefully during the blind test that follows.

If no one hears differences in a sighted warm-up audition, the blind test should not continue.

The anecdotal results from the sighted warm-up auditions are among the most important data from any blind test, but are rarely collected, reported, or even considered, before continuing on to the blind test.

The blind test is unlikely to last more than two to three hours, assuming it follows a one hour sighted warm-up audition -- three or four hours is enough listening for one day -- and maybe too much for some of the listeners who may be fatigued.

If the blind test lasts 160 minutes for 16 trials, and there are 12 listeners, then each participant will average 13.33 minutes
listening in the sweet spot where he can control the A-B switch and play his own music.

That may be enough time to make one good A-B-X decision for one trial out of 16.

But that leaves 15 trials where a participant will NOT be in the sweet spot (assuming a one-person sweet spot typical of two-channel audio), NOT be in control of the A/B switch, and NOT be listening to his own music!

How does that methodology replicate the home listening environment?

What if some of the other blind test participants prefer "machine-gun A/B switching" every five seconds ... or they prefer to listen to the same Barry Manilow song 4 times in a row during their 13.33 minutes in the "sweet spot" ... or even worse, they prefer both "machine-gun switching" AND Barry Manilow songs?

Can the other participants leave the room and complete 15 trials later by themselves?

No, they have to stay and suffer for 146.67 minutes of a 160 minute test when the music and switching are completely out of their control and the sweet spot is occupied by someone else!

So after a total of roughly 220 minutes of listening (60 minutes sighted warm-up audition and 160 minutes blind test) here is what has actually happened:

(1) ABX decisions are hard work and certainly not the same as relaxed pleasurable listening to music at home

(2) Blind tests are directed by a possibly hostile test leader, with an all-components-sound-the-same-agenda, who may even have German ancestors, and who may serve your favorite beer, leading to you into falling asleep while he fills in your scorecard with random guesses if you don't participate ... as compared with relaxed and sober listening to your own music, played on your own stereo, in your own home.

(3) Blind test participants may spend roughly 13.33 minutes in the sweet spot listening to their own music, while spending roughly 146.67 minutes sitting away from the sweet spot, possibly sitting behind someone with a large hat, and/or sitting off center, possibly listening to "machine-gun A/B switching" of Barry Manilow music, or even worse, 146.67 minutes of braying soul divas.

(4) My conclusion about blind tests:
It's amazing there are so few fistfights during and after blind tests!

These so-called "tests" don't even come close to replicating the obviously superior methodology of relaxed long-term listening to your own music in your own home.

That ANYONE ever scores a positive result in ANY blind test is a credit to how well their ears can hear under very trying circumstances ... or perhaps just lucky guessing by particpants eager to get the hell out of there so they can go home to enjoy some music they actually like, on a good stereo!
.

Richard BassNut Greene
Subjective Audiophile 2007

This post is made possible by the generous support of people like you and our sponsors:

Kimber Kable

---

[ Top of Page ] [ Contact Us ] [ Support/Wish List ] [ Copyright © 2024, Audio Asylum, all rights reserved ]

---

[ General ] [ Speakers ] [ Tubes ] [ Vinyl ] [ Digital ] [ Hi-Rez ] [ Video Asylum ] [ Cables ] [ Tweaks/DIY ] [ Music ] [ Films ]

---