We all love a nice testimonial, and they serve a function but how valid might this data be?

When us sciencey-types critically evaluate evidence, we always assess the outcomes measures for validity – what outcome measures are used? What exactly do they measure? How accurately do they measure what the researchers claims they measure? All very important questions. We also look for any factors that potentially bias a measure.

So let’s look at…

Pathway to a testimonial:

· Person chooses to see a therapist

· Self-selection bias, this excludes all the people who do not, or cannot see a therapist so it’s not a representative sample

· Pay money

· Client is invested, wants to see a return.

· Likely believes the therapy will lead to positive outcome

· The belief in a positive outcome is a strong predictor of outcomes actually being positive.

· Has a positive outcome

· Possibility of causation error by client or therapist, non-specific effects of the therapy, natural history (gets better anyway), placebo, regression to the mean – there are a multitude of reasons why a client experiences a positive outcome that may have little or nothing to do with the treatment itself.

· Leaves a positive testimonial

· Seeks confirmation that justifies the cost, confirmation of a wise choice

· Therapist puts it on their website

· Selection bias – only the best make it to the website

By any metric, this is really bad data.

Example:

I invent a made-up therapy and claim it improves fertility (it doesn’t).

Most ignore it, some book in.

Some get pregnant because over time it’s a statistical probability.

Of those, some write a testimonial about how great the therapy is, how knowledgable I am.

I post on the website.

There you have it. Positive testimony for a completely BS therapy.

Recommendation Hyperbole

Has there been a recommendation hyperbole inflation in recent years? We tend not to ‘recommend’ so much as ‘CANNOT RECOMMEND HIGHLY ENOUGH!! Classic eBay seller feedback for someone who sent the thing they said they’d send can yield ‘Excellent service A+++++++++++++++++++’.

Why the enthusiasm? Quite simply, we want people to choose the things we choose. It provides the comforting sense that we make wise choices, that we have good judgement. Perhaps we seek this more so when outcomes are subjective. Have you ever recommended a plumber and caught yourself really pushing the point? I have. ‘Wow, I seem to really be pushing this…’. I also recommend my dentist, but I’ve only seen two dentists for as long as I’ve had teeth, so how could I really know mine is good?

‘Sooooo knowledgeable…’

“It’s quite easy to appear knowledgeable with just a few nuggets of information”

This ubiquitous compliment is also self-serving, unreliable, and can often be found on the most pseudoscientific practitioners’ pages. Can we determine how knowledgeable someone is, if they have knowledge about something that we do not? After all, it’s quite easy to appear knowledgeable with only a few nuggets of information, just takes confidence.

So what does a testimonial really tell us? Who knows, but it probably says more about the client than the therapist. Does it matter? Probably not. But it’s fun to think my fellow dedicated evidence-based practitioners out there may have also felt a bit grubby using them.

This is all without the more cynical view that questions the authenticity of the testimonials, but then with data this bad, who needs to make anything up?

It’s hard not to keep them around though.