How Common Is This?

Lemme start with two disclaimers: 1) this post is super nerdy, and 2) I'm not super online, so I might just be a bit naive here . . .

Yesterday I ran into another audiophile I see from time to time in record stores, and we talked about speakers. I told him about my favorite model (not gonna mention the model cuz I don't want this post to turn into a discussion about speaks), which he said was an "objectively terrible" choice. To support his conclusion, he texted a link to a measurement test report that assessed the speakers to have a particularly bad treble response. He had never tried the speakers, did not look at other test measurement reports, and did not look up subjective reviews. He said subjective data simply didn't matter now that the science is so good. So, his conclusions about the speakers were 1) solely based on objective data, and 2) the data he used to reach such firm conclusions were limited to one test report of one sample by one tester.

(BTW, I was curious, so I looked up other measurement tests of the speakers. I found three other tests, all of which showed his test was an outlier. The treble was flat, which my ears already knew, so he incorrectly concluded the speakers were "objectively terrible." There are a bunch of reasons why his test was an outlier, many of which are completely outside the tester's control. Most likely, the tester just had a bad sample of a speaker.)

Anyway, are a lot of us this hardcore about objective testing (i.e. a product is only going to be good if the measurements say so)? I'm not saying objective data is a bad thing--it's great, and I use it all the time when pre-evaluating products, with the final factor being my subjective experience. But, I feel comfortable enough with analyzing data to for it to be useful. I'm not a scientist, but you don't need to be one to understand the basics of the scientific method for low-stakes decisions like choosing audio equipment. One measurement test is data. Science, OTOH, is the independent, critical review of data, ideally with multiple data sets, to draw conclusions. Of those of us who put tremendous faith in measurement data, how many seem to be familiar enough with the basics of the critical analysis to actually use the data correctly (i.e. do they just look uncritically at a single measurement test of a single sample and consider it valid, settled science)?

Maybe none of this really matters, but if I were a manufacturer of audio equipment, I'd be pissed if a sizable segment of an already niche market was only using one measurement test (often done by an enthusiast for enthusiasts) to write my products off completely.