The NICE Osteoarthritis Guideline Development Group (GDG) has been challenged for not using the available acupuncture evidence in the most appropriate manner in three crucial areas. This response explains the methods used by the GDG with particular reference to the economic analysis, and illustrates that the methods used were those most appropriate for developing a NICE clinical guideline. The cost-effectiveness conclusions made by the GDG are supported by the currently available evidence.
Statistics from Altmetric.com
The linked Debate paper “NICE guideline on osteoarthritis: is it fair to acupuncture? No”,1 presents three issues relating to the way that acupuncture was dealt with in the 2008 National Institute for Health and Clinical Excellence (NICE) osteoarthritis (OA) guideline.2 This paper presents a response to each of the three points. The author writes from the perspective of a health economist who worked on the NICE guideline as a technical member of the guideline development group (GDG), rather than from the perspective of an expert in the literature on acupuncture for OA. The focus will be the health economic methods used to conduct the cost utility analysis of acupuncture presented in the guideline, and how this analysis may have led to the specific recommendations which were developed.
THE LONG-TERM EFFECTIVENESS OF ACUPUNCTURE
The Debate paper states that a meta-analysis of the three existing long-term acupuncture in OA studies shows that acupuncture has a statistically significant (p<0.05) effect on WOMAC pain and function scores compared with placebo.3 This is true, but the meta-analysis also shows no statistically significant short-term improvement in pain and function scores when comparing acupuncture with “true sham”, and demonstrates that there is heterogeneity in the majority of the short-term outcomes, which makes assessment of the true effect by meta-analysis unreliable. Also, another recent meta-analysis of acupuncture for OA of the knee shows that there is a clinically irrelevant short-term and long-term improvement in pain and function compared with a sham control.4 This underlines the issue raised in the guideline concerning heterogeneity of acupuncture trial outcomes.
From a cost-effectiveness perspective, a statistically significant clinical effect does not necessarily mean that the intervention is cost-effective. The three long-term studies included in White et al’s meta-analysis were included in the health economic cost consequence analysis presented in the guideline.5–8 The intervention analysed by Berman (2004) was estimated to be associated with an incremental cost-effectiveness ratio (ICER) of approximately £42,000. The intervention analysed by Witt (2000) was estimated to be associated with an ICER of approximately £10,000, and the intervention analysed by Scharf et al (2005) was estimated to have an ICER of approximately £80,000, all compared with placebo. There was a considerable range in these estimates: one estimate is less than the typical NICE cost-effectiveness threshold of £20,000 per additional quality-adjusted life-year (QALY) gained, and two estimates are substantially higher. The results indicate that the clinical effect shown by the three long-term papers included in White et al’s meta-analysis may not be sufficient to allow the interventions to be classed as cost-effective — based on the three studies the true ICER is likely to lie in the range of £10,000–£80,000, and hence there is considerable uncertainty around this.
White1 suggests that the GDG appears to have interpreted the evidence for acupuncture more conservatively than for other interventions, noting recommendations for topical and oral non-steroidal anti-inflammatory drugs (NSAIDs). A very detailed economic model was developed for NSAIDs in order to update NICE’s previous technology appraisal of these drugs and cyclooxygenase-2 (COX-2) inhibitors.910 The model did not assume a long-term effect of the drugs in question. Regarding adverse events, the data used to populate the economic model were primarily taken from three large randomised controlled trials11–13 which provided follow-up data over 1 year or more. Certainly, evidence regarding topical NSAIDs is less prevalent, but the guideline alludes to a health technology assessment report which has since been published, analysing the effectiveness and cost-effectiveness of oral and topical ibuprofen over a 12–24-month period.14
It should be noted that the NICE OA guideline did not recommend the use of glucosamine and chondroitin preparations, which according to the cost consequence analysis had mean ICERs ranging between £2,500 and being dominated (less effective, more costly) compared with placebo. Hyaluronan injections were not recommended. According to the cost consequence analysis, various types of hyaluronan injection had mean ICERs ranging between £41,000 and being dominated compared with placebo. Based on these data and range of recommendations, it is not at all obvious that acupuncture was treated more conservatively than other interventions included in the guideline.
THE CONCLUSION ON ELECTROACUPUNCTURE
White1 suggests that the conclusions made by the GDG are particularly surprising when it comes to electroacupuncture (EA), and that evidence from the Berman (2004)5 paper and Sangdee (2002)15 paper should have played a part in informing the conclusions made about EA.
As suggested by White,1 classifying these studies as “acupuncture” or “electroacupuncture” is difficult and of debatable importance. As noted above, the Berman analysis was included in the cost consequence analysis, and a mean ICER of approximately £42,000 compared with placebo was estimated. This evidence, as well as that provided by Sangdee et al (2002) suggests that EA is unlikely to be a cost-effective use of resources for the NHS.
White1 argues that acupuncture and EA should be regarded as part of the same therapy. The practice he refers to of using EA as a second-line therapy after manual acupuncture was not evident from the trial designs reviewed, further complicating analyses of these different acupuncture techniques. Beginning with standard acupuncture and moving onto EA if the manual acupuncture was not having the desired effect, would lead to increased intervention costs, but if this was more effective than standard acupuncture alone there may be a possibility of the treatment sequence proving cost-effective. The Guideline group was not aware of published studies which investigated such a sequence and was unable to evaluate this.
COST-EFFECTIVENESS OF ACUPUNCTURE AGAINST “PLACEBO”
White1 identifies two main problems with the economic analysis of acupuncture therapy presented in the NICE guideline. One relates to the use of placebo as a comparator in the analysis, and the second is described as “the inappropriate manner of considering the available evidence”.
Placebo as a comparator
The use of placebo as the comparator is an arguable point, and the GDG discussed this at length. White1 states that there were two main reasons given in the guideline for comparing the interventions included in the cost consequence analysis to placebo:
1. “The comparator should be placebo rather than the next best alternative because often data comparing the next best alternative are not available, and it is not always clear what the next best alternative is.”
2. “… because these interventions are secondary treatments for osteoarthritis rather than core treatments, they generally have very small effect sizes, it may be that a comparison to placebo is optimal.”
The statements quoted by White1 must be taken in context. Only a limited amount of economic analysis can be completed on a NICE clinical guideline — it is not possible to conduct detailed economic analyses into every intervention that has cost implications. To decide what economic analysis is undertaken, the GDG — made up of clinical experts from primary and secondary care, patients and carers — prioritises the areas which it believes are of most importance for further economic analysis, with guidance from the health economist allocated to the guideline. If the guideline is updating any existing technology appraisals, further economic analyses on these specific interventions must be undertaken where relevant. As a result of this process, the main economic analysis included within the OA guideline investigated the cost-effectiveness of oral NSAIDs and COX-2 inhibitors. It was noted that a number of other interventions — including acupuncture — had important resource use implications and therefore the cost consequence analyses were completed. In the introduction to the cost consequence section of the guideline, it was explicitly stated that the results of the analyses should be interpreted with care, and that they do not represent full economic evaluations. The cost consequence table was a method for informing the GDG members of the broad “ball-park” levels of cost-effectiveness of a range of interventions, rather than representing definitive incremental cost-effectiveness ratios; this is reflected in the guideline recommendations and priorities for future research.
The algorithm presented on page 21 of the full guideline shows some of the interventions that may be used by people with OA (education, exercise, weight loss, paracetamol, topical NSAIDs, oral NSAIDs and COX-2 inhibitors, capsaicin, opioids, intra-articular corticosteroid injections, local heat and cold interventions, assistive devices, supports and braces, shock-absorbing shoes or insoles, transcutaneous electric nerve stimulation, manual therapy (manipulation and stretching), and joint arthroplasty). Other potential interventions which were not specifically recommended for use in the NHS by the guideline included glucosamine, chondroitin and combinations and derivations of these, arthroscopy, rubefacients, and hyaluronan injections. Therefore it is very clear that defining one specific next best alternative treatment for a cost-effectiveness comparison is difficult.
As stated by White,1 the NICE guideline to the methods of technology appraisal states that all relevant comparators for the technology being appraised should be included in the analysis.16 It is obvious that including all of these potential treatments in one single economic analysis is not always possible — particularly in the context of a guideline rather than a technology appraisal, which is usually much narrower in scope. An intervention might appear cost-effective compared with one comparator, but not when compared with another. We agree that choosing the “wrong” comparator could lead to decisions which do not allocate NHS resources efficiently. A further reason, not quoted by White,1 was given by the GDG for the choice of placebo as the comparator: “The effectiveness measure is compared with placebo rather than no treatment, as often studies do not include a ‘no treatment’ or ‘usual care’ arm. Comparing with placebo allows more comparability between interventions considered in the guideline”.8 Comparing acupuncture to usual care, but glucosamine to placebo, or hyaluronan injections to saline injections, would not allow a fair comparison of the likely cost-effectiveness of the different interventions. Because all the interventions investigated were compared with placebo within trials, this represented the only “common” comparator.
This does not imply that White’s (2009) arguments are unfounded. It is true that placebo is usually not a relevant treatment option, and therefore comparisons to placebo are not always useful in a cost-effectiveness analysis. Placebo effects may differ for different types of treatment, and may be marked for manual therapies.17 From a theoretical perspective and not doubting White’s assertion of the clinical evidence showing a positive effect of acupuncture compared with sham, the fact that one therapy has a larger placebo effect than another does not necessarily mean that it is a better treatment and that the placebo effect should be incorporated as part of the treatment effect for one intervention but not another. From the perspective taken by White,1 the placebo effect should be included as part of the treatment effect for all treatments, not just manual treatments, as otherwise comparisons between the different interventions would be biased in favour of the manual therapies.
Placed in a position where broad cost-effectiveness estimates were required that would allow some comparison between the numerous interventions included in the cost consequence table (which included other manual therapies such as hyaluronan injections), a comparison to placebo was deemed to be the most useful comparator for inclusion in the analysis.
Inappropriate use of evidence
The second criticism of the economic analysis made by White1 suggests that the evidence for the cost-effectiveness of acupuncture was not used appropriately. The paper by Reinhold et al is outlined as one which should have been given considerable weight in drawing conclusions about acupuncture.18 This was not published at the time of the guideline development, although some data were provided by the authors and a previous paper presented the same results in less detail.19 This earlier report was summarised in the NICE guideline, and it was highlighted that the analysis gave very few details on the costs included and the utility scores obtained and gave cost-effectiveness results only from a societal perspective. NICE is interested in a UK healthcare payer perspective; this study was not set in the UK and cost components and utility scores included in the study were unclear, so it would appear inappropriate to use this paper to form NICE recommendations. As a result, White’s suggested alternative recommendation could not be supported in the guideline; if patients are choosing acupuncture as part of self-care then it is presumably purchased in the private sector, and NICE does not make recommendations for private healthcare providers.
Reinhold et al provide an informative analysis of the cost-effectiveness of acupuncture in a German societal setting, but the following brief analysis illustrates that not basing NICE recommendations on the summarised results of the paper may have been wise.
First, White1 states that the authors assessed the effect over 3 months, and that if these effects lasted 1 year, the treatment would have been more cost-effective. On inspection of Reinhold et al, it is clear that although utility was only measured up until 3 months, the economic analysis then assumed that the gain in utility remained, decreasing at a linear rate, until utility was back to baseline at 12 months. The authors assume that the effects do not disappear until 12 months — at least 9 months after the intervention was stopped. Therefore the quality of life gain estimated by the authors is unlikely to be an underestimate, and if the quality of life gain does not last 12 months the QALY gain will likely be an overestimate, and the ICERs an underestimate.
Second, it is notable that in the 3 months prior to baseline the total overall costs were €686.42 in the control group and €599.65 in the acupuncture group. This difference is not statistically significant, but relatively substantial and could be suggestive of important baseline differences between the study groups. The cost analysis presented by the authors shows that after 3 months the costs in the intervention group were €1,204.15 and in the control group were €734.66, a difference of €469.50. A lower cost difference of €429.86 was used in the cost utility analysis (CUA) as not all patients were included in the CUA due to missing utility data. It could be argued that these cost increases are underestimates, given the difference in costs in the study arms at baseline. The authors state that the cost difference in the intervention group between baseline and 3 months was €604.51, 40.6% higher than the €429.86 used in the CUA. If this €604.51 cost increase had been used in the economic analysis, the ICER for all patients combined would have risen to approximately €25,100.
Third, there is considerable uncertainty shown in the paper’s results. The intervention appears to have a much greater quality of life impact on women than men and the authors note that the reason for this is unclear. The ICERs presented range from being dominant (ie, the intervention is cost saving and quality of life enhancing) for women with OA of the hip, to being €282,602 for men with OA of the knee.
Finally, the authors present cost-effectiveness acceptability curves (CEACs) for nine scenarios (men and women with hip and knee OA individually and combined). These are difficult to interpret due to the lack of a clear distinction between the different lines that are plotted, and the results appear slightly questionable because several of the lines show a relatively high probability of showing cost-effectiveness even at a willingness to pay threshold of zero. Assuming that this is a product of a significant skew in the cost data (though this is not discussed by the authors) and ignoring this issue, the CEACs show considerable uncertainty. At a willingness to pay for an additional QALY threshold of around €25,000 the probability of acupuncture being cost-effective is approximately 85% for women with OA, and approximately 25% for men with OA.
Based on this review of Reinhold et al, it seems likely that had the full paper been published at the time of guideline development the acupuncture recommendations made by the NICE GDG would not have changed. This is a societal analysis and is not set in the UK, but more importantly the results are highly uncertain and the ICERs presented are potentially underestimates. The setting and perspective taken by the study would not necessarily cause the paper to be unused when making NICE recommendations — it is not NICE policy to only use cost-effectiveness data set in the UK, although obviously this is preferable. It is the uncertainty around the results of the paper combined with the non-UK setting that would render it unsuitable as the basis for NICE guidelines.
This discussion illustrates why the work relating to acupuncture on the NICE OA guideline was conducted as it was. It also shows that given the current state of the evidence the conclusions of the economics review would not change now as compared with those made at the time of guideline development, and consequently White’s suggested alternative recommendation would not be supported in the guideline. This is not to say definitively that acupuncture is not cost-effective for the treatment of OA from a UK NHS perspective, but simply that the currently available evidence is not sufficient to suggest that acupuncture represents a relatively cost-effective use of NHS resources. Further research is desirable in this area.
The author gratefully acknowledges the comments on an earlier draft made by Professor Philip Conaghan (Chair of the GDG), Dr Joanne Lord (Technical Advisor at NICE during the development of the OA guideline), Dr Rachel O’Mahony (Health Services Research Fellow in Guideline Development, and member of the GDG), Mr Rob Grant (Project Manager during the development of the OA guideline), and Dr John Dickson (Clinical Advisor to the GDG).
Competing interests: The author was employed as health economist on the Guideline Development Group.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.