Header

Objective Reviews & Commentary - An Engineer's Perspective

May 27, 2011

Cmoy eBay Headphone Amp

ebay cmoy bench with caseVIRTUAL GROUND VS REAL GROUND: I recently tested the popular AMB DIY Mini3 and was disappointed in the performance. The Mini3’s virtual ground was intended to be an improvement on a classic Cmoy but creates serious new problems. I wondered how a classic Cmoy with a conventional ground would compare. See my 3 Channel Virtual Ground article for more.

HERITAGE: Many would argue the humble Cmoy is the headphone amp that started a headphone DIY craze. It was originally designed by Chu Moy and I’m sure tens of thousands have found their way into Altoid tins and other DIY enclosures. It’s similar to the original Grado RA1 headphone amp and is extremely simple. It’s an ideal first time DIY project and there are completed Cmoy amps from a variety of sources—although many deviate significantly from the original such as the Mini3 does.

BUY OR BUILD? I could make a Cmoy from scratch, or use someone else’s PC board, but decided to save time and test a ready-to-rock version complete with a Mini3-like aluminum enclosure. The Cmoy tested here was purchased with Buy It Now on eBay for $39 with free shipping direct from China. It’s a classic dual battery design without any gimmicks or supposed “enhancements”.

FATAL FLAWS & eBAY: Nearly every “no name” audio or DIY product I’ve encountered sold directly out of Asia on eBay has been seriously flawed in one or more ways. This Cmoy was yet another example. I wonder if these flawed products were originally intended to be sold through regular retail channels before someone found the Fatal Flaw. Sometimes the flaws are relatively obvious, as is the case with this Cmoy, and sometimes they only show up if you make the right measurements. But, out of probably 20+ products I’ve encountered, I’d say 90% had at least one significant flaw like this Cmoy does.

NO RETURNS: Would you return a $39 amp to China? I suspect the vendors know few will and eBay's used to “liquidate” flawed audio products--sort of like a factory outlet store selling “irregular” clothing. The flaws are not disclosed, the vendors frequently change their names to dodge poor feedback, and returns are expensive or not allowed. The seller for this amp has disappeared from eBay listings.

ebay cmoy bench setupGAIN? WHAT GAIN? The very definition of “amplifier” is it’s supposed to amplify the input signal—i.e. make it larger and more powerful. This amp’s Fatal Flaw is having no gain. It doesn’t amplify! You put 200 mV in you get 200 mV out even at full volume. In geek speak, that’s called a “buffer” not an “amplifier”. This could still be useful if the source has a relatively high output impedance—this Cmoy’s lower impedance might drive headphones better than not using it at all. See my articles on headphone and amp impedance and headphone amps.

WHO NEEDS GAIN? Say you have an iPod. And, like most iPod’s, it can only manage about 0.5 volts of output. That works great with typical 16 ohm portable headphones but falls short for your 250 ohm full size Sennheiser cans which need more like 2 volts. So, in this case, you need about 4 times or 12+ dB of gain. This Cmoy has 0 dB of gain. The 0.5 volts your iPod can manage is all you’ll get into your Sennheisers. Net gain is zero and the Sennheiser’s don’t play loud enough.

IGNORING THE FLAW (for now): I was curious how this amp would measure ‘as-is’ partly to see if it was even worth spending more time on. The output performance of the op amp was my biggest concern and that’s relatively independent of the modest gain headphone amps need. So I started making measurements and was impressed enough I ran a full set of measurements.

THE AMP: The quality is decent enough and similar to the assembled version of the Mini3. It’s a nice black anodized extruded aluminum enclosure with thumbscrews for easy battery access, metal front and back panels, silkscreening, etc. There’s even a metal volume knob and the volume control feels good. It’s externally much like the AMB Mini3 except there’s no DC power jack for charging the batteries. Unlike the Mini3, this Cmoy has DC blocking capacitors on the inputs to protect your headphones if your source has a DC offset. This also allows the volume control to work better.

HISS & NOISE: Having no gain helps a lot with noise. Even with my SuperFi IEM’s I couldn’t hear any hiss.

SUBJECTIVE SOUND QUALITY: Driving the Cmoy from the headphone output of my Benchmark DAC1 using a variety of headphones it sounded very clean with no obvious flaws of any kind. It did not, of course, make the signal any louder but the Benchmark has plenty of output so that wasn’t a problem.

MEASUREMENT SUMMARY: The Cmoy surprised me. The 4556 op amp doesn’t get much love relative to other more expensive op amps. So I wasn’t expecting amazing numbers even at 150 ohms and I expected it to really stumble into 15 ohms. It’s worth noting the zero gain configuration helps out a bit—especially reducing noise. But even taking that into account, this Cmoy delivered promising performance. It also shows a conventional 2 channel design, even with an inexpensive op amp, can outperform a more elaborate 3 channel or virtual ground design:

Measurement 4556 Cmoy AMB Mini3 FiiO E5
Frequency Response +/- 0.0 dB Excellent +/- 0.1 dB Excellent +/- 0.1 dB Excellent
THD 1 Khz 150 Ohms 0.001% Excellent (1) 0.002% Excellent 0.005% Excellent
THD 1 Khz 15 Ohms 0.003% Excellent (1) 0.017% Good 0.012% Good
THD 20 hz 15 Ohms 0.005% Excellent (1) 0.01% Very Good 0.6% Poor
THD 20 Khz 15 Ohms 0.02% Excellent (1) 0.45% Poor 0.05% Excellent
IMD CCIF 0.003% Excellent (1) 0.043% Fair Not Measured
IMD SMPTE 0.003% Excellent (1) 0.009% Very Good 0.006% Excellent
Noise (ref 400 mV) -96 dB Excellent (1) -94 dB Excellent -86 dB Fair
Max Output 15 Ohms 67 mW Very Good 104 mW Excellent 108 mW Excellent
Max Output 150 Ohms 180 mW Excellent 38 mW Fair 22 mW Fair
Output Impedance 0.1 Ohms Excellent (1) 0.9 Ohms Very Good 0.7 Ohms Very Good
Crosstalk 15 Ohms 68 dB Very Good 40 dB Poor 46 dB Fair
Channel Balance Error  1.1 dB Fair 1.14 dB Fair Sample Problem
  1. These measurements are unrealistic for gains greater than 1X

PLEASE DON’T SHOOT THE MESSENGER: Some might not believe a $39 pre-assembled Cmoy with a $0.50 op amp can measure this good. Before sending me hate mail or posting cranky comments please read my article about subjective vs objective audio. The schematic is posted below for the amp I tested. It should be easy for another engineer to verify these measurements, and indeed, nearly all of them are consistent with the JRC 4556 datasheet.

BOTTOM LINE: I was impressed the 4556 op amp can deliver this kind of performance into low impedance loads. I do, however, need to modify the Cmoy so it has some gain and re-check the appropriate measurements. Check back for those updates. But assuming most of the performance is similar it would be a high performance bargain for someone wanting a portable amp.


TECH SECTION:

CIRCUIT DESIGN: The schematic (from my simulation software) is shown below with the component values of the eBay Cmoy as I received it. The gain is (1 + 470/100000) or pretty much a factor of 1 aka 0 dB or unity. The use of a 47K potentiometer is also not ideal as it will have much higher Johnson Noise than a 10K pot would—but with no gain that doesn’t matter much. For comparison, classic Cmoy schematics can be found at Tangentsoft and Headwize.

eBay 4556 Cmoy Schematic (one channel)

ebay cmoy pcbJRC 4556: This design uses the Japan Radio Company (JRC) NJM4556AD (also called the JRC4556, JR4556, etc.) op amp. This is a $0.50 part and many Cmoy DIYers seem to favor other op amps instead. But the performance, as you’ll see, is more than respectable. Running from a bipolar 15 volt (30 volt total) supply, it’s rated to swing 25 volts peak-to-peak into 150 ohms which is 83 mA peak current. The output current is specified at 70 mA but it can obviously exceed that. It’s specified for audio use with impressively low THD. The 4556 looks decent on paper and even more so on the test bench driving loads it was never intended to drive.

GOLDEN EAR OP AMPS: The preferred op amp for Cmoy designs seems to be the TI/Burr-Brown OPA2134 or OPA134. Tangent has an op amp list that describes trade offs and even supposed sound quality. The 4556 doesn’t even make that lengthy list. Cmoy op amps seem to be sometimes chosen based on heavily biased sighted listening tests and an apparent misunderstanding of what specs matter for audio. The greatest single weakness when using an op amp as a general purpose headphone amp is current capability. The OPA2134 is only rated for half the current output of the 4556—35 mA vs 70 mA. In fact many (most?) op amps on the Tangent list are rated for less than 70 mA.

SLEW RATE MYTH: There’s a myth faster slew rate is highly desirable and some DIY sites and forum members throw around impressive slew rate numbers. But a headphone amp only needs about 1 V/uS of slew rate to handle any realistic signal it will ever see. So a 30+ V/uS op amp is no better than the lowly 4556 rated at 3 V/uS. Faster op amps may perform worse in other areas—especially power consumption which is important in a battery powered device.

SPEED OVERDOSE: It’s easy to understand how a “fast amp” seems desirable but think of it this way: If a single dose of aspirin gets rid of your headache using 100 times that amount is generally a bad idea as the side effects could be lethal. The same is true of ultra fast amps—they’re often less stable, more power hungry and noisier. Best-in-class designs are all about understanding the big picture, what matters most, and balancing the most important parameters and trade-offs. Even with a cost-no-object design, slew rates well beyond what’s needed are often undesirable. This is a long way of saying the 4556 is plenty fast in this application.

4556 VS OPA690 VIRTUAL GROUND: As mentioned above, the 4556 datasheet specifies 25 volts p-p into 150 ohms for 83 mA of peak current. That’s 166 mA total current from both channels. The OPA690 in the Mini3 is rated at 160 mA and shared between the channels. In other words, on paper, the two designs have similar peak current capability. In reality, as you’ll see below, the 4556 in this Cmoy far outperforms the Mini3 in nearly all measurements. This is largely because the Cmoy uses a conventional “real” ground vs the virtual ground (“third channel”) in the Mini3.

NO SERIES OUTPUT RESISTOR NEEDED: The 4556 is supposedly current limited. And, indeed, I tried to blow this one up but failed. It measures just as great after driving a short as before. The series short circuit protection resistor required by the AD8397, and placed inside the feedback loop in the Mini3, is a likely reason for some of its poor performance. The 4556 can be directly connected to the headphones as it is in this design.

CMOY PCB: In the photo you can see the 470 ohm and 100K 1% feedback resistors flanking the JRC 4556. The two 100K resistors above them in the photo provide the input bias for the op amp. The LED dropping resistor is in the top left. Blue boxed film capacitors are used for the input caps (bottom of photo) and decoupling (flanking the 4556). The volume pot is a 9mm enclosed type with a metal cast bearing. The 4556 is soldered in place making “opamp rolling” more difficult.

BATTERY LIFE: The idle quiescent current for the 4556 is rated at 8 mA and that’s exactly what I measured. The power LED, however, only runs from one battery and adds about 2 mA. From a 200 mAH battery this is 20+ hours of battery life at low listening levels and perhaps 10+ hours if you have power hungry headphones and like it loud. This is roughly triple the battery life of the Mini3 as there’s no virtual ground and the slower 4556 consumes less power. The LED is a bit of a design flaw as it may help one battery die before the other by adding 20% more idle drain to one battery—see Uneven Battery Life below.

UNEVEN BATTERY LIFE : Several people, such as the author of this Tangent article, claim dual battery designs may be a poor choice because if one battery runs dead before the other one. The theory is the amp will output DC and may damage your headphones. I discuss this in more detail in the Tech Section of the 3 Channel Virtual Ground article. In testing the theory, this Cmoy still has only 4 mV (negligible) DC offset with one 9 volt battery fully charged and the other breathing its last breath at 1.3 volts. As battery hits 1.2 volts, if it’s the negative battery, the amp just dies with no DC offset. If it’s the positive battery dying the amp will sometimes output serious DC—not a good thing. Long before it gets to 1.2 volts, however, it clips the audio signal badly and sounds terrible. So, in other words, anyone listening would have ample warning it was dying. Even single battery amps can output DC when the battery drops too low.

BATTERY MITIGATION: If I were using this amp on a daily basis, I’d disconnect the LED so the power draw was the same from both batteries, and use brand new alkaline batteries from the same package (or reasonably matched rechargeables). The batteries will more or less die at the same time and any DC problem at the very end when they’re down to the 1.3 volt threshold will quickly finish them off. Your headphones should be relatively safe—especially if you’re listening when it happens. I would , however, suggest not leaving the amp on when you’re not listening to it. But that’s true even of a single battery amp. The only 100% cure is a DC protection circuit.

DC OFFSET: I measured 4.5 mV in both channels which is sufficiently low. Contrary to myth, it doesn’t change with even large battery voltage mismatch (see above).

FREQUENCY RESPONSE: It doesn’t get much better than this. It’s dead flat from 10hz to 48 Khz. The slight 0.1 dB roll off you see at 5 hz are the 2.2 uF input caps. The channel balance is near perfect but with the volume control all the way up and no gain, that’s not surprising (although my FiiO E5 proves otherwise). This is driving 150 ohms to my standard reference output of 400 mV RMS:

eBay 4556 Cmoy Frequency Response 150 Ohms (ref 400 mv RMS)

POWER OUTPUT: Cmoy amps are intended for higher impedance headphones. This one does the job with a very healthy 180 mW into 150 Ohms. By comparison, the Mini3 managed 38 mW into the same load. Into a 15 ohm load it can manage 67 mW--a lot better than most portable players including any iPod or even the Cowon player I tested. Into 600 ohms you would get a still healthy 50 mW. To put this in perspective the popular Sennheiser HD600/650’s need about 2 volts RMS for any sane person and the 4556 with dual 9 volt batteries can deliver over 5 volts (nearly 15 volts peak-to-peak) to those headphones. The 15 ohm distortion is surprisingly low at about 0.003% at lower levels and it stays below 0.008% right up until it clips. The 150 ohm distortion is also seriously low in the 0.0008% range around 1 volt output:

eBay 4556 Cmoy 1 Khz THD vs Output Red=15 ohms Blue=150 ohms Yellow=600 ohms 2X 8.4v NiMh commented

POWER OUTPUT 33 OHMS: Because a Cmoy isn’t designed to drive 16 ohms, I also tested it at 33 ohms where it managed a very respectable 121 mW or roughly the same as the Mini3. And this is with both channels driven:eBay 4556 Cmoy THD vs Output 33 Ohms Both Channels

THD vs FREQUENCY: Here’s the THD+N at 400 mV output into both 150 ohms (yellow) and 15 ohms (blue) versus frequency. The drop above 10 Khz is due to the harmonics falling above the audio band. The performance is excellent into both loads and consistent with the 4556 datasheet:

eBay 4556 Cmoy Frequency vs THD Yellow=150 ohms Blue=15 ohms 400 mV RMS

THD SPECTRUM 150 OHMS: Not much to see here. Both the 2nd and 3rd harmonics are way down at –110 dB. My rule of thumb is anything below –70 dB for the 2nd harmonic and –80 dB for everything else is very likely inaudible. So this clears the bar by a mile:

eBay 4556 Cmoy 1 Khz 400mv In Max Vol 150 Ohms (ref 400 mV RMS)

THD SPECTRUM 15 OHMS: Making it work harder, here’s the result into 15 ohms. The relatively benign 2nd harmonic is still way down at –90 dB and the 3rd harmonic is at nearly 100 dB. This is way better than I was expecting into such a tough load:

eBay 4556 Cmoy 1 Khz 400mv In Max Vol 15 Ohms (ref 400 mV RMS)

RESIDUAL DISTORTION: Here’s the residual distortion waveform at 400 mV out into 15 ohms. It’s clean with no surprises. The 4556 is a very decent op amp even with a 15 ohm load (most op amps are only specified down to 600 ohm loads, the 4556 is specified down to 150 ohms but 15 ohms is another order of magnitude worse):

eBay 4556 Cmoy 1 Khz 400 mV 15 Ohm Residual Distortion

THD+N 20 HZ: Into 15 ohms at 20 hz the Cmoy does very well. The benign 2nd harmonic is just over –90 dB and everything else is below –100 dB: eBay 4556 Cmoy 20 hz THD N 15 Ohms (ref 400 mv RMS)

THD+N 20 KHZ: This is also an excellent performance into 15 ohms (measurement bandwidth to 80 Khz):eBay 4556 Cmoy 20 Khz THD N 15 Ohms (ref 400 mv RMS)

CCIF IMD: Another great performance into 15 ohms. Everything in the audio band is down around –90 dB or better:

eBay 4556 Cmoy CCIF IMD 15 Ohms (ref 400 mV RMS)

SMPTE IMD: Again, great performance. The sidebands around the 7 Khz signal are below –100 dB:

eBay 4556 Cmoy SMPTE IMD 15 Ohms (ref 400 mV RMS)

INTERNCHANNEL IMD: The channel shown is driving 0.9 volts at 1 Khz into 15 ohms. And the other channel is driving 0.9 volts at 300 hz into 15 ohms. I normally run this at 1 volt but that’s clipping for the Cmoy, so I had to lower it slightly to 0.9 volts. The result here is further proof how much better a “real ground” performs than a virtual ground. If you look at this test on the AMB Mini3 the result was far worse (see 2nd graph below). Here there’s mostly just the expected 300 hz crosstalk and the expected harmonics of the 1 Khz signal. There’s almost no actual IMD despite this being a Class A/B op amp and all the power supply ripple and “ground contamination” the 3 channel proponents claim are a problem:

eBay 4556 Cmoy 1 Khz 900 mV Interchannel IMD 15 ohms (300 hz not shown ref 0.9 V)

MINI3 INTERCHANNEL IMD FOR REFERENCE: Compare the Mini3’s virtual ground design to the result above and note the much higher intermodulation products and dense “forest” of distortion created by the shared artificial ground:

AMB X-Audio Mini3 THD N 1 Khz Left 300 hz Right 1 Volt RMS 15 Ohms (ref 1 V) comments

VOLUME & GAIN: This Cmoy, oddly, has unity gain with the volume all the way up. In other words it has no voltgae gain. This is the design’s Fatal Flaw:

eBay 4556 Cmoy 1 Khz 400 mV in Max Volume 100K Load (ref 400 mV RMS)

OUTPUT IMPEDANCE: With a 100K “load: the voltage was 401.4 mV (see above) and dropped to 398.5 mV with a 15 ohm load. This works out to an extremely low output impedance of 0.1 ohms (which is at least partly my cables, the jack, etc.). The reason it’s so low is there’s no series output resistor or capacitor between the op amp and the headphone jack and the op amp is operating at maximum feedback. This is as close to ideal as it usually gets.

CHANNEL SEPARATION (CROSSTALK): Again, this shows how much better a real ground performs compared to a virtual ground. This is excellent crosstalk performance and is in the range of just the output jack itself. The ground layout on the PCB helped here:

eBay 4556 Cmoy 1 Khz Crosstalk White=15 ohms Blue=150 ohms

NOISE: The noise performance at half volume was excellent. But there’s a reason—no voltage gain. Most headphone amps have at least 8 - 12 dB of gain but this one has none. It shows the 4556 itself is very quiet:

eBay Cmoy 4556 Noise Vol=50% ref 400 mV RMS

CHANNEL BALANCE: The worst case channel balance was, as is often the case, at –45 dB--the lowest volume setting I measure at. It was only slightly over 1 dB which is very respectable and typical of the better analog pots (NuForce should take notes here!):

eBay 4556 Cmoy Volume & Channel Balance Worst Case (ref 400 mV RMS max vol)

PHASE: Like the Mini3, the Cmoy had essentially perfect phase performance:

eBay 4556 Cmoy Phase Response

SQUARE WAVE RESPONSE: Driving the reactive load of real 16 ohm headphones (Sennheiser CX300’s) the Cmoy had only a tiny bit of overshoot and no significant ringing. The tiny peak could be removed by adding some compensation to the feedback loop. Overall this is excellent performance:

eBay 4556 Cmoy Square Wave CX300

SLEW RATE: The large signal performance is shown below and the slew rate measured 3.8 V/uS which exceeds the worst-case 4556 spec of 3 V/us. The Cmoy only needs a 1.2 V/uS slew rate so this is plenty fast enough:

ebay 4556 Cmoy Square Wave & Slew

THE FINE PRINT: The measurements, unless otherwise noted, were made at max volume which is unity gain. The batteries were 8.4 volt Ni-Mh rechargeables and measured 9.0 volts while running the amplifier. All measurements were consistent with my more recent reviews and the methods outlined in my Testing Methods article.

TECH SECTION SUMMARY: As I said in the first part of the review, I was impressed how well the 4556 works as a headphone amp. Considering the measurements are basically of the 4556 itself, with little besides potentially the PCB layout and batteries to hinder its performance, most of this shouldn’t be a huge surprise. But I was surprised the power output matched the Mini3 into 33 ohms and the distortion was so low into 15 ohms. Whoever designed this Cmoy PCB did a good job with the layout, grounding, etc. The measurements clearly show the superiority of a real ground vs a virtual or “3 channel” ground. I need to correct the gain flaw and conduct some follow up measurements, but consider the 3 channel/virtual ground myth further busted. And don’t underestimate a lowly $0.50  op amp or Cmoy!

May 18, 2011

Subjective vs Objective Debate

argument by Francis CarnaubaINTRO: Given the strong reaction to some of my articles here are some thought’s on what’s likely behind the more emotional responses. The world of high-end audio can be almost religious and divided something like the Republicans and Democrats. In this case it’s the Subjectivists versus the Objectivists. It’s been called the “Great Debate” a “Holy War” and more. (photo: Francis Carnauba)

HIGH-END AUDIO’s DIRTY SECRET: What if I told you there was a proven way to evaluate gear using your own ears that highlights even the smallest audible differences between two pieces of gear? Compared to typical listening methods it’s been found to be far more reliable and revealing of subtle differences. More on this later!

THE SUBJECTIVISTS: The hardcore Subjectivists trust their own ears above all else and often ignore, downplay, or sometimes even actively discredit objective efforts. Some argue they have superior hearing and/or listening skills and more refined tastes. That sometimes creates at least a whiff of an elitist “club” that some are drawn to (think Robb Report). But, regardless, their genuine passion for audio is to be admired. And I believe at least some of them do have superior listening skills compared to the Average Joe. Despite their more emotional left brains, which might imply a greater love of music, there’s some consensus Subjectivists spend more of their time tweaking and evaluating their hardware than a typical objectivist. Stereophile’s Michael Fremer is generally considered a strong subjectivist.

THE OBJECTIVISTS: This group tends to prefer some sort of science, measurements, or objective listening tests to back up claims of “A is better than B”. When reading a gear review they’re more likely to skip to the measurements section (if there is one) than read subjective impressions. They tend to be skeptical of outrageous claims and ultra high priced gear. They also tend to buy less expensive gear, less often, than subjectivists making them less attractive to manufactures. As mentioned above, they tend to be more satisfied with their systems so the spend more time just listening to music rather than the gear. Some have speculated this is because they’re confident more of their hardware is already “good enough.” Peter Aczel and the late Julian Hirsch are classic audio objectivists. And a lot of the folks at Hydrogenaudio fall in this category.

THE MODERATES: Just as with politics and religion, it’s not black and white. Some have a foot firmly in both the objective and subjective side of things. Some examples are John Atkinson at Stereophile, John Siau at Benchmark Media, and to some degree, myself. We value objective measurements but also trust our ears and just because we may not hear a difference we accept someone else might. I believe those in the middle are generally the most open minded.

wine heatheronhertravelsCREDIBILITY: Wine critics need credibility and trusted taste buds to discern all the subtle details of wine. Subjective audio reviewers are expected to have good hearing and highly developed listening skills. But objective geeks only have to make proper measurements others can verify. Even Grandpa with his hearing aids could do it. (photo: heatheronhertravels)

ACCOUNTABILITY: The subjective reviewers have it easy. If someone doesn’t agree with one of their reviews, excuses are plentiful. When subjective reviewers are questioned, I have seriously seen or heard variations of all of these responses:

  • While you might not like it I preferred the slightly more recessed presentation of the UberDAC Black Edition
  • The UberDAC is a better match with my ultra expensive reference system than your more modest gear
  • Did you use the UberLink Reverse Twisted Unobtanium cables I recommended for the UberDAC? 
  • I was in a noisy restaurant for lunch immediately before reviewing the UberDAC and my ears hadn’t fully recovered
  • I didn’t know it at the time, but it turns out I was in the early stages of a head cold when I reviewed the UberDAC
  • I had too much wine the night before (my personal favorite)

OBJECTIVE ACCOUNTABILITY: Objective reviewers have it hard. We either publish reasonably accurate measurements or we get caught with our pants down. The whole idea is to publish numbers in a way someone else can reproduce (or come close enough). So the quality of our measurements determines our credibility. An error is an error. It requires a lot of wine before 1+1 = 3. We can’t simply ignore or change the well established principals of audio engineering. We have to admit when we’re wrong or join the Planet Earth Is Flat Society.

JUST MESSENGERS: Independent objective reviewers, me included, just test gear and publish the numbers. We don’t make the gear, we usually don’t make up the measurements, and we certainly better not make up the results. We didn’t invent THD, the decibel or Ohm’s Law. So the numbers are what they are. If we’re doing our job right, we’re just messengers delivering numbers we have little control over. Of course nothing but numbers is boring and only useful to hardcore geeks and engineers. So it’s best when the numbers are presented in a relatively understandable way.

YACA (Yet Another Car Analogy): Say you’re in the market for a new fast car and you’re comparing them online. Chevy and Ford both have all new versions of the Camaro SS and Mustang GT. They’re so new nobody has done any track testing yet but here are the factory’s published numbers:

Specification Ford Mustang GT Chevy Camaro SS
Curb Weight 3605 Pounds 3860 Pounds
Horsepower 412 HP 426 HP
Torque 390 ft-lbs 420 ft-lbs
Performance 0-60 MPH 4.8 seconds 4.9 seconds
Gas Mileage (city) 18 miles/gal 16 miles/gal

Based on the numbers, the Mustang is lighter, quicker and uses less gas so you check it out at the dealer. It looks great and seems to have enough power so you buy it. When you get home you find the latest issue of Road & Track in your mailbox. They just tested the your Mustang GT and here’s what they found versus Ford’s numbers:

Specification Ford’s Spec R&T’s Measurement
Curb Weight 3605 Pounds 3910 Pounds
Horsepower 412 HP 290 HP (on dyno)
Torque 390 ft-lbs 275 ft-lbs (on dyno)
Performance 0-60 MPH 4.8 seconds 9.1 seconds (on track)
Gas Mileage (city) 18 miles/gal 14 miles/gal (test loop)

mustang 500MARKETING MEETS REALITY: It turns out Ford’s marketing team wasn’t even close to accurate. The car is way heavier, has a lot less power, drinks more gas, and a Prius with a full charge might give you a good run at a stoplight. This is what you just paid $35,000 for? Faced with the bad news, here are some possible options: (photo: Ford Motor Company)

  • Take the car back to the dealer, show them the article, and ask what’s going on
  • No longer trust Ford and buy a Chevy 
  • Offer your buddy with the dynamometer a case of beer if he’ll test your Mustang’s horsepower and torque
  • Try a few of your own 0-60 runs to see if it’s really closer to 9.1 seconds than 4.8 seconds
  • Burn the issue of Road & Track as you don’t really care about numbers anyway
  • Fire off an angry email to Road & Track accusing them of being incompetent without ever trying to verify if their measurements are even correct

BUGS ON THE WINDSHIELD: If we treat cars like audio, it seems most who already own the Mustang prefer the last choice above along with some or all of the following thrown in for good measure:

  • Shoot The Messenger! He’s clearly an idiot!
  • I trust Ford is less biased than some guy at a magazine!
  • His V8 Mustang was only running on 5 cylinders!
  • He can’t tell the big hand from the little hand on his stopwatch!
  • Those bugs on the windshield were slowing it down!

snake-oil_thumb2

CARS vs AUDIO: Of course you never see automotive numbers off by the huge margins shown above. Why not? Because magazines like Road & Track keep the car manufactures honest. If they know their cars will be track tested, dyno tested, etc. it’s in their best interest to publish reasonably accurate data. If they didn’t it would be obvious. So why should audio companies be different or exempt from being held similarly accountable? (photo: dbaldwin)

FLAK JACKET REQUIRED: Not only is audio myth-busting a relatively thankless job, it sometimes requires protection from an angry mob of subjectivists. Some wonder why I’m relatively anonymous. First of all, it’s recommended Bloggers write under a pen name. And it’s a good thing as I’ve been threatened, called all sorts of names, accused of having other agendas, and much more. Apparently it’s dangerous work being a messenger, reporting real numbers, and challenging audio claims with real engineering! Isn’t Ford responsible for publishing misleading numbers? Why would someone attack Road & Track for helping expose the truth? The same is true of industry standard PC benchmarks. If the Dell claims their laptop is faster than the competing HP, but it’s really the other way around on a dozen different tests, would you attack the guy doing the review? There’s something odd going on with audio.

IT GETS PERSONAL: There’s a long history of attacking objectivists who try to clarify what matters and/or bust audio myths. Peter Aczel of the Audio Critic is a long time example and a newer one is Meyer and Moran for their SACD hi-res audio work. Unlike cars or PCs, much of high-end audio has little basis in fact. When someone tries to bring facts into the mix, some take it as a challenge to their personal beliefs, personal hobby, etc. And, unfortunately, some go on the defensive and try to discredit the messenger. It’s not too far removed from creationists attacking the science behind evolution.

FOLLOW THE MONEY: They say to figure out what’s really going in our messed up political system you just have to follow the money. It turns out, you can largely do the same thing in high-end audio. A lot of money gets spent based on highly biased subjective evaluations of audio gear. In fact, the more expensive the gear, the more likely it’s bought entirely based on subjective criteria. I show an example below in Subjective Report Cards. If you look at what most influences buying decisions--websites, magazines, the largest forums, etc.--you’ll find nearly all of them are largely bought and paid for by the companies making the gear. So it’s hardly surprising few do objective testing. And what objective tests they conduct often give the equipment the benefit of doubt. A classic example are A/V receiver tests where the manufacture’s power claims are rarely directly challenged. Instead they typically run a couple of power tests done in such a way to not highlight the fact a $1000 “120 watt x 7” receiver might manage only 28 watts/ch with all 7 channels operating.

BlindVsSightedMeanLoudspeakerRatingsSUBJECTIVE BIAS: Some interesting studies have been done about subjective bias in audio. Tom Nousaine published a 1991 AES paper titled Can You Trust Your Ears? It included several different tests, but one of the more interesting involved listeners evaluating (unknown to them) identical musical selections. They were asked if they preferred A, B or had no preference. 76% of them expressed a preference despite the selections being identical. It showed people readily hear differences when none exist. That’s not good news for someone who just replaced their $300 DAC with a $3000 DAC because they thought the more expensive one sounded better. The two may really sound the same. For more on this I recommend: Dishonesty of Sighted Listening Tests by Sean Olive. (photo: Sean Olive)

HEARING BIAS: There’s an entertaining and excellent AES Audio Myths Workshop Video with some very interesting observations by experts in their respective fields. One talks about how the brain and ear work together. Human's are not like an ideal microphone treating everything the same. As with the rest of our senses, the brain adapts our hearing for the needs of the moment. If you’re trying to pick out a conversation across a crowded room you naturally filter out everyone else talking without even realizing you’re doing so. This same filtering goes on when listening to audio gear. You might listen critically to the bass one moment and the highs the next, but you can’t listen to both at once and glean as much information. We do the same thing with our vision. There are televised examples of naked people running across football fields in plain view during key plays and few people ever even saw them. Their senses were heavily biased toward the players. They also talk in the video about replacing the circuitry of a high-end audiophile amplifier with a really low-end amp that no audiophile would be caught dead listening to. And, guess what, audiophiles still loved the amp because their eyes were telling them it was a high-end amp so that’s what their ears heard.

bbc mcgurkINVOLUNTARY BIAS (added 5/23): Subjective audiophiles often claim they’re not affected by the sort of sighted listening bias documented by Toole & Olive. They argue bias is only an issue for untrained listeners. But what if it’s genuinely involuntary? There’s a well understood phenomena called the McGurk Effect. It shows how certain knowledge, such as what we see, influences what we hear. And, even more significant, the bias is involuntary. Around the 2 minute mark in the video linked below they talk about how, even when your conscious brain knows what the truth is, your subconscious brain still alters your hearing in ways you cannot control. The researcher says he’s been studying this effect for 25 years and it still affects him just as much as an untrained listener. The McGurk effect goes away if you close your eyes. But in evaluating gear just closing your eyes isn’t enough if your brain still knows what you’re listening to. You need a blind test to eliminate the bias. Try it yourself with this fascinating video: (photo: BBC)

FOLLOW THE MONEY PART 2: It’s human nature if you go out and spend your hard earned cash on some new piece of gear you want it to be worth the investment. This feeling is compounded by all the subjective reviews you read where others raved about the same piece of gear. In multiple ways your subconscious is already wired to hear a nice improvement even when there isn’t any improvement. This isn’t far removed for hearing “Fa” when the guy in the video above is clearly saying “Ba”. Your brain and senses are just telling you what they think you want to hear.

fiio e5 test

THE NEED TO BELIEVE: There are some negative comments in my Mini3 review saying I lost all credibility by comparing it to the $20 FiiO E5. If you look at the measurements, the two amps are relatively similar in many areas. But there are obviously some who need to believe a $180 amp is a lot better than a $20 one. They probably already have a Mini3 so it’s an insult for someone (me in this case) to indirectly suggest they spent many times more than necessary. I received similar comments for suggesting the $29 Behringer UCA202 measurements were respectable—very likely from people who spent a lot more for their USB DAC. Again, in their need to believe, these gear owners would rather shoot the messenger. It’s a kind of denial.

photo by <a href="http://laughingsquid.com">Scott Beale / Laughing Squid</a><br /><br />This photo is licensed under a Creative Commons license. If you use this photo within the terms of the license or make special arrangements to use the photo, please list the photo credit as "Scott Beale / Laughing Squid" and link the credit to http://laughingsquid.com.JUDGMENT DAY: Tens of thousands of “believers” launched massive campaigns to warn of “May 21st 2011 Judgment Day”. The world was supposed to start self destructing with massive earthquakes and more. Scientists tried to point out the facts but the believers didn’t want to hear any of it. If you’re reading those masses of fierce believers were obviously wrong. Even in 2011 lots of people fall under the spell of others, give in to peer pressure, myth, etc. They’re surprisingly willing to believe things with little or no basis in fact. Once these beliefs reach a critical mass, those within the group are very difficult to persuade they might be wrong. But they don’t have a good track record. From those who believed the earth was flat, to those who thought it was going to self destruct in 2011, the science geeks are the ones who are far more often correct. (photo: Scott Beal/Laughing Squid)

BELIEVING = BIAS: If the guy who thinks I’m an idiot for comparing the Mini3 to the E5 were to sit down and listen to both side-by-side which do you think he’d say sounds better? There’s almost zero chance he’d choose the E5. This same bias is widespread in audio. You have a $300 DAC, you arrange to listen to a $3000 DAC, and even if they sound exactly the same, your brain and hearing are “wired” to think the $3000 DAC sounds better. So how do we get around this problem?

BED SHEETS & TESTING: As you probably guessed, blind testing is the “dirty secret” I referred to at the start of the article. Matrix Audio conducted a relatively simple and eye opening example. The photo at the right shows the test set up with two different systems under a bed sheet sharing a pair of high-end speakers. Volunteers stood behind the speakers and swapped the high-end cables. There were no switch boxes involved. The result, if you haven’t seen it elsewhere, is the listeners couldn’t tell a high-end $12,000 stack of gear from a $700 (I’d say closer to $400) set up with a pro-sound power amp, bargain basement CD player, and a cheap obscenely long RCA cable connecting the two. You can read all about it here: (photo: Matrix Audio)

LONG TERM LISTENING: A lot of blind testing involves switching between A and B, or replaying music tracks after something is changed. Critics of these tests argue that’s not the best way to evaluate audio gear. They say you must live with it for a while to appreciate the differences (never mind most of them claim to swap out a piece of gear and hear immediate and obvious differences). David Clark and Laurence Greenhill came up with a clever idea. They made a bunch of sealed black boxes where some had a direct connection inside while others distorted the audio signal to a significant degree. They were built with high-end connectors, etc. They sent the boxes home with members of a local audiophile club to live with and decide if they had a “straight wire” box or one that did ugly things to the audio. Despite living with them for a while, the audiophiles who took the boxes home failed to determine which was which. The same boxes, however, were identified with relative ease in a blind A/B/X test. This demonstrated the exact opposite of what many audiophiles claim: Long term listening is less sensitive than A/B/X testing. This test, and others, are summarized in Ten Years of ABX Testing.

A NEW WINE ANALOGY: Many have probably heard the analogy before, but here’s a short entertaining article by a wine critic describing blind testing. He rated the $2.50 Charles Shaw wine very poorly in sighted tasting. But he discovered, with brown paper wrappers on the bottles, it was “not going to be easy”. I won’t spoil the outcome but his experience is exactly analogous to what usually happens in blind audio testing. Suddenly that $20 FiiO or $30 Behringer is a lot harder to pick out. At least this critic was humble enough to go public with his experience:

blind listening tests nwavguyBLIND BANNED: The largest headphone forum around, Head-Fi, prohibits the discussion of blind testing in all but one of their 20+ forums. They only, seemingly grudgingly, allow it in the back-of-the-bus Sound Science forum that’s all but ignored by the mainstream. Why? Could it be their many sponsors, say Qables selling iPod cables priced at many times the iPod itself ($1000+ for a 6 inch iPod dock cable!), don’t approve of having their products debunked with blind test results? You won’t see many blind tests in the audiophile magazines or on ad-supported websites. It seems an excellent tool has been strategically marginalized, swept under the rug, and discredited over the last decade by the “industry”. It’s time for more people to start asking why.

SUBJECTIVE HAS ITS PLACE: When choosing a car, new laptop, and audio gear, the subjective side matters. It’s not all numbers. Things like ease of use, aesthetics, and build quality are all important. When it comes to the sound of different speakers and headphones, subjective opinions are often what matter most. Are you a basshead, like it bright and detailed, laid back, or as accurate as possible? I’m not trying to dispute subjective preferences. If you like the sound of tube gear, even if it measures poorly, that’s your business. If it puts a smile on your face that’s what matters most. And some buy high-end gear for the quality, looks, status, etc. My concern is misleading objective data, objective claims with no basis in reality, marketing “pseudo-science”, and when the line between subjective and objective is intentionally blurred.

SUBJECTIVE REPORT CARDS: Stereophile magazine assigns letter grades in their annual Recommended Components issue. In the April 2011 issue the Vitus Audio MP-P201 phono preamp, for a paltry $60,000.00, rated an “A+” while the boring $199 NAD PP-3 rated a sorry “D”—the kid who didn’t study for the test. Stereophile is to be commended for conducting measurements of some of the gear they review. In this case, the bargain NAD measured better than the uber-expensive Vitus. The NAD had notably lower distortion and lower noise—an especially important parameter in a phono preamp. So is it safe to assume the extra $59,801.00 of value in the Vitus must be purely subjective?

WHEN OPINION BECOMES FACT: In the example above one person, Michael Fremer, apparently decided the $60K Vitus was vastly better based on his personal opinion of how it sounds. From what I gather, he didn’t conduct any sort of rigorous blind listening tests that included others. Nor did John Atkinson’s inferior measurements ultimately carry much weight. I gather Fremer listened to the Vitus in his particular system, using his phono cartridge(s), with his particular tastes in music, and decided it’s worth the astronomical price tag. Is the rest of the world to believe they would also prefer the Vitus over the NAD even with their different personal preferences, phono cartridge(s), and music collections? This often happens at all price levels—someone else’s highly subjective (and nearly always biased) opinion becomes objective “proof” that Gear X is better than Gear Y. So lots of other people buy Gear X even though they might have different tastes or not hear any difference at all. There are many things wrong with this—especially when seemingly objective ratings like Stereophile’s  “A”, “B”, etc. or another magazine’s 5 stars, imply some clear criteria.

PEER PRESSURE: The Stereophile ratings are a lot like Wine Spectator scores. They create a sort of “peer pressure”--much like wanting to serve wine with a high score regardless of the wine buyer’s own personal tastes. And even if a critic enjoyed a particular wine with his Italian food, it might be a lousy match with someone else’s Sushi. The same can be said for Michael Fremer’s reviews. Just because the Vitus sounded good for him, using his gear, music, etc., doesn’t mean it’s audio nirvana for someone else. And how much was Fremer unavoidably biased by the $60K price tag and similar factors? Did someone from Vitus fill his head with hyperbole beforehand over a gourmet lunch? Perhaps most serious of all: What if the Vitus really sounds just like the cheap but well engineered NAD? That outcome is far more likely than most realize or want to admit.

BACK TO ACCOUNTABILITY: If Michael Fremer can go around recommending $60,000 gear based on his listening abilities, some might reasonably want proof he’s qualified. And being a good sport, Mr Fremer broke rank and participated in at least a few blind listening tests. The result was a rather mixed bag and at least one included lots of hand waving. Not surprisingly, few want to follow in his footsteps. Today it’s even more difficult to find listeners with a public reputation willing to participate in a blind test. The same people who publish hearing “immediate and obvious” differences in everything from cables to power conditioners typically make all sorts of questionable excuses when asked to do so with brown bags or bed sheets concealing the gear. Personally, I suspect Fremer probably can hear things 99% of the population would have trouble hearing. If anything, he’s a “ringer” for the subjectivists and I have genuine respect for his listening abilities. So it’s especially a shame he, and other skilled critics like him, won’t participate in more blind tests.

rca cablesWIRED WISDOM (updated 6/3): Tom Nousaine published a great article in Sound and Vision called Wired Wisdom. The goal was to see if audiophiles, in their own homes using their own familiar high-end systems, could hear differences between cheap and expensive cables. In all three trials, they could not. The cable myth suffered a serious blow from reality. The second link compares expensive versus cheap speaker cables with similar results:

CREATIONISM vs EVOLUTION: Alan Lofft, the editor of Sound and Vision, tried to dance around the Wired Wisdom article--likely to appease their cable advertisers. He talked of creationists and evolution arguing both had an important role. And he compared high-end cables to “audio jewelry” that some buy for aesthetics and status rather than sound quality. Lofft did what much of the high-end audio industry does. He tried to soften the truth, not offend too many people, and make sure the status quo (along with their advertisers) remained relatively unscathed. See: Follow The Money. His column is on the last page of the Wired Wisdom article. I’m getting flak for not performing a similar delicate dance with AMB, NuForce, etc.. It seems nobody is supposed to rock the boat too much however factual their concerns. Is this audio or political foreign relations?

HIGH RES (SACD) vs CD: Multiple tests have been published comparing standard 16 bit 44.1 Khz CD quality audio to higher resolution formats such as 24 bit 96 Khz and SACD. The most famous is probably this one:

Meyer and Moran played SACD content with the ability to switch an A/D –> D/A pair operating at 16 bits and 44 Khz into the signal path. In other words, the high resolution SACD audio was sometimes “down converted” to CD quality. They designed the test to give the listeners “every opportunity” to detect a difference. The testing lasted a year and included 60 members of the Boston Audio Society, many professional recording engineers, fresh eared college students, and a whopping 554 listening trials. After all that, the only way anyone could identify a consistent difference was by cranking the volume unrealistically high during quiet passages exposing the higher noise floor of the 16 bit conversion.

Think about the implications of the above. Most subjective audiophiles claim to hear differences between CD players, DACs, and indeed most anything that performs a digital to analog conversion. They also consider SACD and other high resolution formats as being plainly superior. Why can’t audiophiles detect any difference at all when the music is subjected to an extra A/D and then another extra D/A process when they don’t know that’s happening?

SHOOT THE MESSENGER (again): Not surprisingly, many have tried to discredit Meyer and Moran claiming they didn’t use the right source material, etc. The study authors have responded to much of the criticism in this little cited follow up. It’s my personal opinion nobody over the last 4 years, despite plenty of attempts and desire, has invalidated the overall results. If nothing else the study demonstrates just how genuinely transparent 16/44 digital audio can be. The best challenge I’ve seen is a single 2010 study finding a tiny minority of expert listeners, under very specific circumstances, could discern very slight differences. There are many other interesting references about SACD vs CD as well as a summary of the above test here:

THINK ABOUT IT: Two guys come along and the most respected objective audio organization in the world publishes their paper that threatens to destroy the entire SACD audio industry and also does damage to high resolution audio formats of any kind. Meyer and Moran attempted to demonstrate CD quality audio really is good enough. If Philips, Sony and the music labels behind SACD knew the study was flawed, and SACD was audibly superior, it would have been pocket change for them to fund a study demonstrating where Meyer and Moran were wrong. But, surprise surprise, that never happened. The closest was the 2010 paper on Sampling Rate Discrimination mentioned above (if you want to geek out on a lengthy discussion of the 2010 study check out this Hydrogenaudio thread).

RINSE LATHER AND REPEAT WITH VINYL: Lots of claims are made for the analog nature of vinyl LPs and a small fortune is spent on esoteric turntables and phono gear as an analog source for pure analog high-end systems. When playing vintage analog-mastered music the audio never suffers the indignity of being whacked up into a bunch of numerical values and put back together. Much like the SACD test above, there have been various tests demonstrating even devout vinyl lovers can’t tell when you slip an A/D –> D/A loop into their otherwise all analog signal chain. Here’s a link to one but there are some better ones I’ll work on finding the links for. I’ve also done my own informal blind vinyl test on the sly. The vinyl lover wasn’t even aware he’d been listening to digital for several days on an extremely high-end all analog system. And this guy really hates digital anything. Don’t get me wrong, I own a nice turntable and I listen to vinyl. But for me it’s mostly about music that’s only available on vinyl. I don’t pretend it’s a technically superior format.

OBJECTIVE ISN’T EVERYTHING: I said above subjective stuff matters, and I’m including the reverse just to be clear. The numbers only tell part of the story. They make a convenient way to compare some things—especially say power output, output impedance, how suitable a given source/amp is for a particular headphone, frequency response, etc. But there are limits. If nothing else, great measurements provide a significant piece of mind for some people. They can relax and enjoy the music knowing their gear is among the most transparent available. And blind tests “wrap” subjective listening in a controlled, and more objective, environment free of the usual bias while keeping score. The two complement each other well.

WHEN SPECS ARE NOT ENOUGH:  First the easy part. For speakers, headphone and phono cartridges I think everyone agrees it’s tough to look at the specs and know exactly what they will sound like. You can still make some valid comparisons but the specs only give you a partial idea of the sound. With speakers and headphones the acoustics are a big part of the listening experience—all rooms and ear/head geometries are different. The sound of cartridges are altered by the tonearm and turntable geometry they’re used in (effective arm length, VTA, arm resonance, damping, etc.).  They also perform very differently playing worn vinyl as stylus tips come in an almost endless variety of shapes and sizes. So cartridge A rides in a different part of the groove than cartridge B. And measurements are limited by the relatively low resolution of vinyl test albums.  So, in other words, your mileage may vary and caveat emptor. You have to listen to speakers, headphones and cartridges to fully evaluate them. But that’s much less true with electronics.

GRAY AREA (added 5/31): There’s a solid consensus on the subjective nature of speakers, headphones and phono cartridges but what about everything else? Most objectivists will tell you a $20 well designed interconnect and a $200 well designed interconnect will sound the same. And that’s been demonstrated many times (see Wired Wisdom above). Hardcore objectivists (such as Peter Aczel) argue any amp that measures sufficiently well and is operated well within its limits will be indistinguishable from any other amp. And that’s been demonstrated in countless blind listening tests. But what about when an amp nears its limits? What if you have difficult to drive speakers for example that are 2 ohms at some frequencies? Will the cheap amp still sound just like the high-end model? Perhaps not. Some of these behaviors can at least be partly measured but some are more difficult. And what defines “measures sufficiently well”? Enough studies have been done it’s fairly safe to make several generalizations, but there’s still room for discussion and further research in some areas. I’ll hopefully be publishing a future blog article on the topic of correlating specs with listening observations.

MAGICAL THINKING: As someone pointed out in the comments, some audiophiles are in this for the “mysticism, magical thinking, and never ending quest” (their words) and I certainly know a few in that category. The purveyors of tube products tend to be rather clever in their marketing. They rarely make boastful performance claims and some offer hardly any specs at all. They know certain people enjoy their products and they make an appropriately subjective sales pitch. That’s hard to argue with. But when someone makes objective claims, and they’re far from being realistic, that’s just deceptive and wrong—regardless of the buyer’s priorities. It’s easy for some to get “sucked in” to the hype, myths, and mania when the real facts are constantly being swept under the rug. It’s not unlike Wine Spectator scores. See Peer Pressure above.

MORE INFO: If you’re seeking more interesting info check out:

TO BE CONTINUED… I have lots more to share on this topic, but for now, I’ll see what sort of response this brings.