hahaha, nice.

Yeah bad for business

Its exactly this. People that make money on selling stuff dont like the results of abx testing because they loose money. We in the sound science and recording industry love it because we like to debunk the myths and snake oil crap.

You are correct. The whole area is replete with those who prefer to use the power of suggestion. There are, of course, subject aspects to music; it's an art form, but the same is not true when it comes to the technical accuracy of reproduction. I manufacturer X claims their extremely expensive loudspeaker cable is better than generic cable (of an appropriate gauge), then there's no reason why that can't be double-blind tested. It doesn't even need to sound better, just different would be a start. The same then goes for a whole host of things, like audio quality power cables, or digital signal cables, or network switches. That is not to say that there might not be strong preferences for those things that can actually make a difference, but in those cases, a double-blind test would show that people hear there is actually one. But for many components this is not the case, and particularly not for those that do simple, functional roles rather than, say, act as transducers.

i'm a total cheapskate so rigorous testing has value to me. The idea of spending a few hundreds of dollars extra on something I can't even hear in a controlled environment is pretty concerning.

mic drop, well said.

You are just writing a lot of nonsenses. Paul very eloquently describes why it’s useless and why A/B blind test works fine. As he says it works for pharmaceuticals but that’s something else.

@@dieselbrodeur Two years and this is your response? (that the vested interest of an MD selling overpriced kit knows better than the whole scientific community). It’s almost as if these companies rely heavily on snake oil because they have no peer reviewed scientific ABX evidence and cant scientifically prove their claims - if they did publish such results it would be damaging to their business model. Buy hey you believe Paul has perfect pitch, can hear a pin drop from 50metres away and the golden ears of a dog. A 60-80 year old biological marvel defying the laws of human aging, if you will… deary me.

@@dieselbrodeur The real reason why they do not ABX test is because they cannot objectively demonstrate their subjective claims. Its almost as if there is a business model that relies heavily on snake oil to justify the price tags. Overpriced cables, mains conditioners and plethora of other items conjured up are a complete and utter con. In the beginning I was also conned into spending many thousands on kit that made no objective difference (rather, it made me feel better at the time). I say this as a public service announcement for you: Wake up you fool.

haha!

Exactly ! Once upon a time a sells person even came to my house , cause I asked him if we can test his super expensive speaker cables . He agreed to blind test , and guess what I was playing my old very inexpensive cables and his , switching in between . That blinded fool guy said , that my inexpensive speakers cables sounds super good . He was so stupidly sure it was his cables , haha .

Then that would be a blind test? If you weren't blindfolded then you would be subject to expectation bias.

@@zulumax1 That was my point, yes

shhhhhh, such things are too hard to handle! :D

​@@harackmw, not only that, but such things make it hard for equipment makers to get you spending $$$ on stuff that either doesn't do anything (power conditioners, LOL), or does it no better than other stuff that is 5 times cheaper.

@@VioletGiraffe Yup, or those insanely priced cables.

i've done it myself and with some friends, years ago when i was still into audiophile stuff. basically, dac, 0 - 0.5% difference. amps, most listeners could actually hear a difference, but most of the time can't tell which one is better and nearly 50%/50% would end up 'preferring' the cheap entry level ones or the premium ones. cable? well i didn't even bother embarrassing myself for trying

Well, apparently audiophiles can't stand to have their golden ears tested, they're too sensitive.

They actually play behind a screen so that the selection panel is unaware if one of the musicians is personally known to any individual on the panel. Appearance does come into play later when they want to make sure the musician blends with the other musicians and does not distract from the ensemble and performance.

@@raulgarcia1718 Apparently it was also a problem of knowing what the person was playing, as in what type of instrument and make, all things that people judge others on.

yup.

The likelihood of being able to achieve true double-blind testing when it comes to audio equipment is virtually zero. It works for medicine (with the additional element of the placebo to make it gold standard) but it is impossible to replicate for audio equipment. People bandy the term "double-blind" about without understanding that double-blind means that nobody - not one person - in the chain knows which is which (in medical tests that means the patients AND the doctor are all unaware what they are given/taking) - and that makes it practically impossible for implementation outside of medicine. Wine tasting is never double-blind - it is always blind tasting, meaning that the participants are unaware what they are tasting but the host does (before you ask, 15 years as a wine merchant before a change in career - thousands of wines tasted and many, many long evenings of blind tastings). If any audio company made the claim that it was conducting double-blind testing I would call them out on it.

@@richardt3371 Good point, lets make it a single-blind test! Doesn't change the fact that some "blindness" in the process would help exclude snake oil form real value. That's what an open minded reviewer thinks about blind reviews: kzfaq.info/get/bejne/p7CnrJio3JjGe3U.html (minute 4:35).

If you play your guitar just by your ears and brain and don’t play with emotional feeling, most people will be way less interested in what you are playing..

@@nicktube3904 We are not talking about putting emotion into a musical performance, but about taking emotions out of a listening experience so as not to be unduly biased when listening to and evaluating different equipment. I wonder what would happen if you took two speakers, A and B, one costs $500, the other $1,500. You display the stickers on the speakers and have people come in to listen/audition the speakers. Well, I would guess most people are going to think the $1,500 speakers sound better. Now... I wonder what would happen if you switched the price tags/stickers and bring in new people to listen, you think most people are going to think the $500(really $1,500 speaker) is better? My guess would be many would not(the speakers tagged with the $1,500 sticker but really only $500 would sound better to them) -- cognitive bias at its best. If I were listening to a speaker/amp/preamp, CD player, DAC, cables, whatever, I would want to make an objective decision, the best one I possibly could, and so I would want to remove whatever subjective biases and preconceived notions I may have about whatever piece of equipment I'm hearing. More importantly, I would think the one that sounds better to me would also be the one that moves me more emotionally, and price/brand/looks should not bias that perception.

@@mydogskips2 my English isn’t that good, but yes it’s good to be objective. But for me personally I choose my gear 49% on ‘technical’ sounding good to my taste and 51% how emotional I got evolved by gear. I agree with you that expensive gear is currently not also the best sounding. The funny thing is that in practice (And I have listens to allot of gear, brands cheap and ridiculously expensive) until to a certain budget it is always a compromise. Emotional engagement vs technical good sounding. For example tonal rich, wide soundstage, organic human voices which are emotional engaging already for me.. VS really high resolution, lots of detail and razor sharp imaging.. In the end I don’t understand what people like about this analytical sound, I have never heard a Cello in a razor sharp image. In real life the frequencys and tones are room filling, that’s why I prefer gear like Audio Note!

Music without emotion is sound and that is not what high end audio is all about

​@@edoreumerman4537 I'm talking about evaluating pieces of equipment, not listening to music. You have to differentiate between those two. And Hi end audio is all about sound. If your really listening to music, the gear doesn't matter. A music lover uses gear to listen to music, an audiophile uses music to listen to gear.

If you want to use an analogy, I always thought cars work well. For the average consumer, a car gets you from A to B and reliability is key, but for the enthusiast the drive is part of the experience. Those are the people who will go out and purchase a luxury car. Some prefer big muscle cars, some prefer very sleek and well beautiful cars, and some prefer cars that has great handling, etc. Then there are the few high end enthusiasts who will go purchase a Lamborghini, Ferrari, or a Bugatti. So when Paul says ABX testing doesn't work, it's because while you can measure how fast a car goes you can't measure how much excitement or joy it brings because it's subjective. However, to the individual its easy to determine whether you prefer one car over another when you get to test-drive both. Same with audio.

@M Pi Well like with wines there is also an element of training involved. And double blind testing can fool most except a few talented winos. JBL has another take on blind testing of speakers I understand.

@@dan.nathan The same goes for bicycles and here’s another analogy, the setting up of a bicycle is as important as that of a hifi system.

I'm sure a lot of high end equipment out there is the same design that well-made middle-end products have, but in a prettier box with better marketing.

i know youll enjoy your DirectStream DAC..... it is incredible!

@@googoo-gjoob But it is only $6000, I clearly need a $10000 dac

@@harackmw , that tongue in your cheek..... does it hurt? i own the DS DAC. it truly is incredible. 30 day in home.... nothing to risk.

@@googoo-gjoob The risk is trying a blind test and failing to spot the dac. :D

@@harackmw that would be scary. when i swapped out my Musical Fidelity dac for a DirectStream dac..... i wouldve been able to tell the difference even if i was wearing earplugs. it was the single biggest jump ive ever experienced. _s t a g g e r i n g_

"it also makes it easy to imagine a difference when there isn't any." That isn't really relevant. In a properly conducted double-blind test with sufficient samples, then it is easy to work out if any differences claimed are statistically significant or essentially random. That's assuming that it is properly set up of course (and that's not always as easy as it sounds). As for the 5 second, or 10 second or whatever wait time is required, then the audiophile industry is always trying to tell us that there are dramatic differences, often described using some ill-defined terms, like brilliance or clarity. If the difference is so clear cut, those few seconds won't make any difference. The same is true of clinical trials. Those in the test and control groups are often asked very subjective questions, such as how much pain they are in, or whether they slept well, or what side-effects they experienced. It's only after crunching the numbers you work out if there's a real pattern or not.

@@TheEulerID You are very right on all points. I know that for me a 5 second silence makes it pretty much impossible to tell a difference between two pieces of well-sounding equipment (not broken or just horrible). Unless it's speakers, the difference between different speakers is always large enough. And yes, a blind test does remove the problem of imagining things (not even necessarily double blind).

Exactly! But if we remove that very modest improvement from High End Mftrs, they would all go broke. That is what they sell is that minuscule advancement (which sometimes isn’t any advancement in sound, but only in appearance).

Agreed. But subtle here, subtle there, subtle everywhere and it adds up. Worth it or not lies in the capability of the buyer, we are no one tell them that it is not worth it or it is too expensive. Me personally? I cant afford expensive shit like that.

Congratulations, you proved to yourself that your ears, your system or your room are not capable of resolving the difference. Therefore, you can save money, by not purchasing more expensive cables. What you have not even come close to proving is that more expensive cables don't make a difference. Surprise Surprise

@@markwagner1997 how should my ears be able to tell a difference if expensive measurement gear can’t? Good cables for 50 bucks are as transparent as super expensive cables.

Took and entire day to do a very controlled ABx test on a single $10,000 speaker cable vs 14ga lamp cord It was really freaking hard. The group of 20 "Golden Ear" Audiophiles was completely spent by the end of the day. Conclusion. For the most part we could not decisively tell which one was which. Length of wire was a couple meters.

@@juliangst measurements only show what can be measured by a specific test, not how a cable can effect sound. For my ears and the level of resolution my system achieves, 100 - 150 dollar Zu speaker cables and Morrow or Canare interconnects made a noticeable improvement. I borrowed some high end cables ($800-$1000) on a couple occasions and once again, my ears and my system did not find any improvement, so I happily did not feel any need to spend more. My experience is not representative of all people's ears or systems, nor is it a judgment on all cables.

@@wilcalint People who buy those expensive cables want to hear a difference so they claim stuff like ‚this cable has more low end‘ or ‚this cable is brighter‘ but in reality all of those cables don’t alter the signal by the slightest amount.

What is a placebo audio component? Because if I have someone tie my CD player to my amp with an interlink and a piece of rope made to look like an interlink, I will definitely be able to tell the difference through which one I am listening.

@@D1N02 A rope is not a placebo for an improved cable; it's just not a cable. It's the same as giving people rat poison in a clinical trial and call it a 'placebo'. I guess an example of a placebo audio component would be a cheap cable painted gold with a high selling price. The gold paint and the high price are the placebo. The "placebo effect" that Paul touched upon wrt clinical tests is that our subjectivity (thinking we actually took a working medicine) can actually have a positive effect on our healing process. This is indeed a real thing when testing medicine, but also completely irrelevant in audio testing. Our subjectivity has no effect on audio gear as it's not part of our body. And yet, salesmen of audio snake oil products always depend on the placebo effect. Subjectively they will absolutely work if you believe in them. It will all sound so much more "revealing", "3D", so much more detailed and realistic! But objectively they do nothing. That's their business model. And I believe the truth is that many of those audio snake oil salesmen actually fell into their own trap; they actually believe what they're selling actually works when in reality they don't do anything.

I wouldn't call it placebo effect, but confirmation bias. You spend a lot of money on something, so it has to be good.

You can't have a placebo in an audio test. What do you suggest - don't play music and mime?

@@richardt3371 Are you deliberately missing the point?

Great observation. Similarly for hi res audio file formats. When we listen to music, we only listen to one copy, not constantly doing A/B with two file formats of the same recording. If the difference takes strenuous A/B testing by trained testers to discern, it's unlikely to be noticed during single copy listening of music. It doesn't matter for enjoying music at all.

Same here

Exactly. This will make some people puke but I love the sound of tape and also the sound of properly crafted FM radio over many so-called benchmark formats. Call me weird.

@@InsideOfMyOwnMind Remember when FM radio stations had their own live studios? No wow and flutter there and I would record it on a hi-fi VHS tape which is still analog with almost no wow and flutter. Only a few good FM stations left that have decent sound, limiters turned up too high, no dynamics.

It doesn't matter whether the person is biased towards the equipment if they can't actually see or otherwise know what they're listening to, that's the whole idea of blind testing.

And so is blind AB

And that is going to depend a lot on the set up of the testing hardware.

We are not speaking of speakers here nor are their beliefs that the same applies to speakers; speakers definatly sound differently no one is claiming otherwise.

ABx testing is VERY HARD. The closer the A & B are the more difficult it is to tell if x is either A or B. It can take hours and hours and by that time your completely spent.

That 10% is imperceptible by 90% of human ears. And 99.99% of ears above 40 years old.

Think of it like going to an exam - many people underperform or fumble at exams, because they get nervous as fight/flight kicks in. Guessing it some degree of that he wishes to avoid. (not gonna argue with it, just explaining it).

You’re convinced that science doesn’t work by a non-scientist?

@@philiprobar I was convinced that Paul has a process that works.

​@@Wacoal34dthat both works and sells

I find power cables and interconnect plays a much bigger role.

@@allansh828 I don't agree with you on the power cable thing as our group did an ABx test on that one. Result, we could not decipher A or B from x. But the interconnects there was some Golden Ears who could successfully pick correct 70/30. Not great but some success.

@@wilcalint Bill, while I agree with most of what you’re saying, I’d say don’t waste too much more of your breath on this line of discussion.

@@allansh828 I find the air density to play a very emotional role.

@@fonkenful I was quite facinating at the time. And a lot of energy was put into it by the group.

This has to be the first time that anybody quoted Joe and it not being a joke or was it 🤔😉

Hah, hah, hah!

Paul is just a salesman

No, he objects to a scientific test conducted unscientifically. You simply can't eliminate emotional response, or tastes, or preferences, from a blind test of audio equipment. ABX testing is inherently flawed where it involves humans expressing an opinion. You cannot take the emotion out of subjective listening, blind or otherwise. Data-driven testing is a completely different thing.

@@richardt3371 One can easily test for emotional preference. Which joke is more funny? Ask 100 people, and you will get an answer. Or the infamous Coke vs. Pepsi test in neutralized containers. You can see whether people can tell the difference, and then state a preference. And you can evaluate that statistically. No problem. The same applies to audio. Can you hear a difference between two cables/DACs, etc. and which one sounds better to you. The problem is, that most people cannot hear a difference, and that is a problem for sales.

@@danielgeiger7739 Sigh. You're missing the whole point of blind testing, which is designed to eliminate preconceptions or tastes, and which is why it is of no use whatsoever as soon as you introduce emotion. Asking someone if they prefer Coke or Pepsi, and testing them blind, gives you no useable information - you can't "evaluate that statistically", what you end up evaluating is human response, not whether one is actually better than another. It's an interesting experiment but it's also dead data.

@@richardt3371 OK: statistics 101. First you can statistically figure out whether one person can distinguish between Coke and Pepsi. Give them as much time/drink as they want to familiarize themselves with each, then do the blind test where they need to assign the sample to either coke or Pepsi.> 8/10 correct, they pass. Re preference, that is a n =1 issue, of course. So you get 10 (or more) people who can distinguish the two, then determine their preference. Then you see whether there is a population preference for Coke/Pepsi. If there is a >= 8 vs 2 distribution, then there is a population preference. Same can be done for cables, DACS etc. If it is

FLAC is PCM audio. Do you mean DSD vs PCM?

That’s an interesting way is saying that you don’t understand psychology testing.

The reality and the sad part is the DSD component may very well be better suited to those other set of components than the PCM component is. Too many variables. One needs to test the whole setup and see if it works for him/her/it/they and the room it is played in.

It's called snake oil salesman

Agreed!

You'd be right if your sample size is only 10 people, but what if it's 10,000 or 100,000? Not that that's realistic, but if it were theoretically performed and 5,000 out of 10,000 could not hear a difference over not just one track but 20 tracks, then it could be the difference is irrelevant to the human ear and not worth the money. Statistics is not a perfect science, but it can give you a degree of confidence when your sample is big enough

@@Chessmapling if 5000 out of 10000 people could not hear a difference, that means 5000 more COULD. Putting these actual humans under 'chance' is just ridiculous. Statistics are useful but dumb when used this way.

The blind bit it crucial, it removes bias. Any non-blind testing is biased, even if you don't realize it (usually you don't!).

Not at all. If you have a properly designed switching system so that A/B/X are clickless and quickly switched (time aligned, level aligned, etc) then you allow the use of short-term loudness memory, which is shown to be the most sensitive analysis of any kind of perception involving partial loudnesses.

@@jamesjohnston3643 Loudness should be exactly the same. Clickless is indeed better. All this has nothing to do with whether it is A/B or ABX.

@@AALavdas I refer you to the term "partial loudnesses" not "loudness", in other words, the detected loudness across the basilar membrane as a function of frequency and time. With a proper ABX, one can most easily detect the DIFFERENT pair, by comparing A with X and B with X. This is still a pairwise comparison in a sense, the key being (as always) to listen for difference, rather than similarity. A/B only offers you "same" or "not same", and does not give you the permanent A and B references (A and B should not change identity throughout a single test series, although they can move in another test series) and by doing so, removes the sensory anchor that makes hearing the difference an easier task.

@@jamesjohnston3643 Well, saying "across the basilar membrane" does not add anything to this sentence. And there is no way to know that, unless you have a probe on the basilar membrane. I agree that there are different merits in different approaches, but blind A/B is fine.

@@AALavdas Well, I didn't come here for an answer, but I didn't expect to encounter somebody who denies settled science. I would suggest that you check the works of Zwislocki, Greenwood, Zwicker, Allen (Jont, not Jon), Hall, and others as far as being able to estimate, at various levels, the excitation on the basilar membrane. You are completely incorrect in your dismissal. Furthermore, you might learn what "partial loudnesses" are. I can create two very different signals that have the exact same loudness (in sones, phons, whatever you want) that will have entirely different partial loudnesses. Allen's work on self-training tests shows, conclusively, that proximate, clickless comparison at the subject's will provide the smallest dl's.

Cool story bro.

@@Frankonero12 Thanks brother. As I said, I'm brand new to the channel, and _I am loving these Q&A's!_

Now there's budding NASA engineer in the making. I just "feel" this circuit is better. Would you bet your life on that?

That is exactly what ABx testing is NOT. And that's where Paul is wrong.

So if a system's Music stirs your emotions , that is not a good thing ?

@@larrywe3320 that's the point it mucks with your objectiveness when evaluating. Great when your just enjoying music but not when doing a/b testing. Unless that is your basis for evaluation.

Well, but he just ended his explanation saying he uses blind AB testing, but not ABX. Scientific methods are quite useful in industrial environment. At home you use your emotions, in a factory you test the hell out of it.

Blind AB is absolutely necessary, and that's what he does. He does not do ABX