Statistics from Altmetric.com
The National Institute for Health and Clinical Excellence (NICE) finds it difficult to understand acupuncture controls. As readers may recall, we were somewhat exercised by the NICE guidelines on osteoarthritis in 2008.1–3 NICE calculated the cost-effectiveness of acupuncture by comparing it with sham acupuncture. We argue that cost-effectiveness comparisons are only useful in making a decision about treatment if they compare treatments that are actually available. Therefore, they should have compared acupuncture with usual care, as they did with back pain.2 Recently, Latimer and colleagues4 showed what a dramatic difference it makes which control you select: acupuncture for osteoarthritis is clearly cost-effective against usual care, but not against sham acupuncture.
NICE has done a similar thing in their headache guidelines, though with less serious results.5 It used a ‘network meta-analysis (NMA)’ to compare the efficacy and cost-effectiveness of five different prophylactic treatments of migraine. When treatments A and B have not been compared directly in a randomised controlled trial (RCT), an NMA is a way of comparing them by using the results of trials of A and B against a common control group, usually placebo control. NICE assumed that ‘placebo’ was a common control for drugs and acupuncture, overlooking the fact that ‘placebo’ acupuncture is not inert, like placebo medication. That is why it is referred to as sham acupuncture.
NICE calculated that topiramate showed an effect size (median difference −1.02) almost twice that of acupuncture (−0.58).
To demonstrate this error, we used NICE's own data for the response rate (percentage reduction in days with migraine) of the placebo/sham groups for the five treatments that were compared. (In the case of angiotensin receptor blockers we had to consult the original report.) Readers can see from figure 1 that the response to sham acupuncture (43%) is considerably larger than that of placebo topiramate (25.5%) and the other drug placebos.
Evidence from more conventional studies supports our contention that NICE's use of NMA is unreliable. For example, the median difference for acupuncture (−0.58) was no different from that of propranolol (−0.58). Yet the Cochrane review found acupuncture was better than β blockers in direct comparisons (two studies, 564 participants, −0.66 (−1.18 to −0.13 )).5 Similarly for topiramate: the calculated median difference for topiramate was greater (−1.02) than for acupuncture, but RCTs by a single research group seem to suggest the treatments are at least similar, or possibly acupuncture is better. The responder rate for topiramate (100 mg) was 37%; topiramate (200 mg) 35%; placebo 22%6; but in the acupuncture study, the responder rate for acupuncture was 47% and sham acupuncture 39%.7
Acupuncture is a sensory stimulation technique, so sham acupuncture must be considered differently from placebo drugs. The inconsistency in control arm response rates is a fundamental challenge to NMA: the lesson seems to be that authors using NMA should test placebo response rates for heterogeneity.
Fortunately for patients, NICE considered the evidence for acupuncture was sufficient to recommend it for both tension-type headache and for migraine. The message for acupuncture researchers is that a study comparing it directly with topiramate is overdue.
Competing interests Both authors are employed by the professional organisation—the British Medical Acupuncture Society, are editors of Acupuncture in Medicine and have received fees, travel expenses and royalties in relation to acupuncture.
Provenance and peer review Commissioned; internally peer reviewed.