Objectives To assess two aspects of the external validity of acupuncture research for osteoarthritis knee pain and determine the common acupoints and treatment parameters used.
Methods The external validity of 16 randomised controlled trials (RCTs) was investigated using a scale consisting of two aspects: reporting and performance. The reporting aspect included acupuncturist's background, study location, treatment detailed, patient characteristics, positive trial results, adverse effects and between-group statistical differences, whereas treatment appropriateness, appropriate controls and outcomes were classified as the performance aspect. Acupuncture treatment in RCTs was compared with common practice according to the literature sources and survey of acupuncturists working in different parts of Thailand.
Results The levels of external validity for the reporting and performance aspects were in the range of 31.3% to 100%. Statistic values such as mean difference and confidence interval were reported by the minority of trials (43.8%). Patient satisfaction and quality of life were seldom used (31.3%). There were minor differences between research and practice in terms of the points used (25.0%), number of treatment sessions (6.3%) and frequency (12.5%). The most frequently used points were ST34, ST35, ST36, SP6, SP9, SP10, GB34, Xiyan and ah shi points, and the commonly used treatment parameters were 20 minutes, 10–15 sessions and two treatments weekly.
Conclusions Reporting of the external validity of acupuncture RCTs for knee pain was notably inadequate in terms of trial setting, treatment provider and statistical reporting. The majority of studies involved appropriate controls and outcomes and applied acupuncture treatments in line with practice.
Statistics from Altmetric.com
Well conducted randomised controlled trials (RCTs) are accepted as the gold standard for evaluating medical interventions.1 ,2 Published trials must be transparent to allow critical analysis in order to evaluate the strength of the data.3 The reporting of clinical trials becomes an important issue when making a judgment on the trial quality.4 External validity can only be assessed with full reporting of trial information, and including the setting.5 Several terms for external validity—generalisability, applicability, transferability and extrapolation—are used, with overlapping meanings.6 Despite the fact that studies with high methodological quality receive more weight in the systematic review results when establishing the overall value of an intervention,7 to enable the clinician to apply the intervention to any patient population, the issue of external validity is critical.5 ,6 ,8,–,10
The determinants of external validity of RCTs are numerous.5 ,11 The external validity criteria have been classified based on the threats which affect trial applicability.12 Some studies evaluated the reported information of trials using one or two criteria,9 ,13 ,14 while several studies suggested a list of criteria in evaluating the trial reports.5 ,6 ,8 ,15 Researchers and editors have put efforts into improving the reporting of the results of RCTs in acupuncture with the main focus centring on Standards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA) guidelines.16 ,17 These follow the Consolidated Standards for Reporting Trials (CONSORT)18 format acting as an extension of the CONSORT guidelines for the specific requirements of acupuncture studies. In fact, external validity has increasingly become an issue of concern and has been raised by several acupuncture researchers.19,–,23 Miao23 conducted a systematic review to determine the external validity of the RCTs of acupuncture for low back pain and found major problems in the external validity of all cases. Ahmad et al15 conducted a review and reported inadequate reporting data related to external validity specifically from RCTs assessing pharmacological and non-pharmacological treatments for hip and knee osteoarthritis (OA). Purepong24 systematically reviewed the detailed information of acupuncture intervention for lower back pain (LBP) according to the six items of STRICTA guidelines and proposed appropriate treatment regimens based on the details of the positive RCTs. Vas and White25 conducted a review and proposed data for the treatment for OA knee pain. However, the data available to indicate the quality of external validity on research regarding acupuncture intervention was inadequate. With an increase in RCTs over the years as well as the different approaches of searching and data configuration, this study identified research papers on acupuncture for OA knee pain during the period 1973–2011 to assess the external validity in terms of reporting and performance aspects and also to provide detailed information regarding acupuncture treatment as an alternative protocol.
RCTs that investigated the effect/effectiveness of acupuncture treatment using the individual points chosen based on the acupuncturist's discretion or the standard set of acupuncture points for OA knee pain were searched for using the following keywords: acupuncture, knee pain, OA knee, OA knee, osteoarthritic knee, randomised controlled trial, randomised controlled trial, controlled clinical trial. Four computerised databases were searched: PubMed (1973 to November 2011), Scopus (1975 to November 2011), Science Direct (1984 to November 2011), Cochrane Controlled Trials Register (Central/CCTR) and the Physiotherapy Evidence Database (PEDro). The references given in these papers were also examined and relevant articles were identified by citation tracking. RCTs were included if they had parallel or crossover designs with comparative treatment groups (either, other therapeutic treatment, no treatment, or acupuncture treatment with different techniques). All studies whose participants had knee OA were included. All papers included were published in English.
Characteristics of included studies
Data related to the design of studies, patient ages, intervention, sample size, outcome measures and the statistical results were presented and summarised as tabulated data. Methodological score was determined using the scale from PEDro,26 the Physiotherapy Evidence Database, to indicate the methodological quality of each study.
External validity assessment
The external validity checklist was modified based on the concepts of the CONSORT statements,18 STRICTA guidelines,16 ,17 the criteria used by Rothwell5 and the applicability assessment scale.12 The checklist comprised two parts: reporting and performance aspects. Each item also contained subsections.
The details of the trial procedure relating to external validity were grouped into five categories:
The practitioner's background and the place of trial setting.
The eligibility of recruited participants in terms of age, duration and severity of disorder.
Acupuncture intervention such as style, points, treatment parameter and technique and cointervention.
Reporting of between-group statistical significance and the adverse effects of acupuncture.
Statistical results such as mean, SD, mean difference and CI.
Three items relating to methodological aspects were used:
The difference between trial protocol and routine practice was determined by comparison with either literature sources (26 textbooks) or practice (11 acupuncturists). Studies that applied unsupported treatment points and parameters (at least one) were deemed to be non-realistic protocols.
The main outcome(s) chosen were considered in terms of whether they were patient centred (such as pain, disability, quality of life, global perceived effect).
Appropriate control interventions were assessed according to Birch27 as to whether they were appropriate to the research aim(s). Comparison with conventional treatment should aim to assess the effectiveness of acupuncture or additional acupuncture; while using no treatment control should aim to assess the (common) effect of acupuncture and could inform researchers whether conditions were improved if without any treatment.
Comparison of acupuncture treatment in RCTs with literature and practice
A total of 26 textbooks (online Appendix 1) were examined to explore the consistency of the treatment selection. A broad sampling of literature was conducted including a compilation of materials written or translated into Thai/English based on traditional Chinese medicine (TCM) diagnosis and treatment. Of the total 26 acupuncture textbooks, 21 were English composed of basic TCM knowledge and/or Western medical knowledge combined with TCM. The remaining five textbooks were written in Thai and identified the acupuncture point locations and functions including indications for the pain condition or disease.
A survey among a sample (n=11) of Thai doctors/acupuncturists (2–30 years clinical experience) working in hospitals located in northern (n=1), eastern (n=2) and central (n= 8) parts of Thailand was conducted. Information regarding the acupuncture points and treatment parameters applied for knee OA were obtained from the survey sample. Eight of the sampled group had received training from China and three had trained and qualified with certificates of acupuncture from the Ministry of Public Health, Thailand.
The points cited in RCTs supported by the literature and acupuncturists were collated to provide information about the use of acupuncture for knee pain. Three treatment parameters (treatment time, number of sessions and frequency) for each RCT were compared with the range of use from the literature and acupuncturist practice. The commonly used points and treatment parameters in the RCTs were analysed as an indication of clinical appropriateness.
Description of the RCTs
A total of 16 studies were located using a computerised database search and included in this review (online Appendix 2). The study characteristics of the included studies were extracted and are described in table 1 and summarised as percentages in table 2.
Assessment of external validity
The percentages of RCTs which reported/performed the applicability of each criterion are shown in table 3.
Patient ages and severity of disorder were well defined (100% of RCTs). Slightly under half of the trials (6/16) reported all items of acupuncture treatment criteria, while descriptions of cointervention, needle type, depths of insertion and response elicited were reported less frequently. In all, 10 studies reported some minor adverse effects after treatment such as bruising, dizziness, discomfort or pain during or shortly after treatment, and no adverse effects appeared in 6 studies. The statistical values reported for the mean difference and CI appeared to be low. A total of 14 of 16 studies (87.5) applying appropriate control(s) matched research question(s).
Differences of treatment protocol between RCTs and practice
In total, of the 30 acupuncture points found from RCTs, 22 points (73.3%) were supported by either 1 literature source or acupuncturist (table 4).
The remaining eight points (ST6, SP4, GB41, BL20, BL58, BL62, LR3 and TE5) were not used and rarely applied in RCTs. The most common use of retention time was 20 min mentioned by 9 of 10 textbooks (90.0%) as detailed in table 5.
The most commonly used number of treatment sessions by RCTs and also that most frequently recommended by acupuncturists and textbooks were 10–15 (85.7% and 66.7%, respectively). The most commonly applied frequency in RCTs was twice per week (mentioned in 4 of 7 textbooks (57.1%) and 8 of 11 acupuncturists' recommendations (72.8%)).
Applicability of the treatment protocol
The nine most common points used (at least 30% of 16 RCTs) were ST34, ST35, ST36, SP6, SP9, SP10, GB34, Xiyan and ah shi points. In all, 10–15 sessions were most frequently applied as reported by 50% of RCTs. The needle retention time should be 20 min (reported by 66.7% of RCTs) and the appropriate frequency two times weekly (reported by 62.5% of RCTs).
This review systematically assessed the quality of reporting and performance in OA knee pain conditions involving acupuncture trials during 1973–2011. The results indicated that the level of external validity of the included RCTs for each item ranged between 31.3 to 100%. None of the RCTs provided the external validity details for all reporting criteria. There were minor differences between research and practice in terms of points and treatment parameters. The nine frequently used points were ST34, ST35, ST36, SP6, SP9, SP10, GB34, Xiyan and ah shi points. The most commonly used treatment parameters were 20 min for needle retention time, 10–15 sessions and two times/week.
In terms of external validity, the details of intervention given to patients were obtained from the reporting of the referral source/treatment setting and acupuncturist background. Reporting the referral source/treatment, such as from hospital or from private clinic, informed the data relating to the treatment techniques given in different ways. For example, practitioners who provided acupuncture treatment in hospitals29 and the acupuncturists who practiced acupuncture in oriental medicine hospital30 or integrative medicine clinic36 might employ different techniques of acupuncture treatment such as applying the diagnosis based on traditional Chinese medicine or Western medicine, using the standardised or individual protocol or applying the specific cotreatment technique (such as moxibustion, cupping, bloodletting puncture). Otherwise, the differences in the severity of the disorder could be attributed to the trial settings/referral sources of the patients. For example, Witt et al34 recruited patients seeking help from a general practitioner, which were very different from those on a waiting list for surgery in Williamson et al's study.33To provide further intervention details, the level of training or experience of the acupuncturists should be reported to inform the reliability of the treatment given to patients.44 In particular, the acupuncturists who participated in the trials should report their training (eg, training in the standard of treatment to be given, the use of the sham devices in acupuncture treatment vs sham acupuncture studies), in order to guarantee the qualifications and professional expertise behind the intervention being explored in terms of its effectiveness or efficacy.44 Providing explicit details about who delivered the acupuncture intervention could help establish a greater degree of reliability surrounding the intervention, especially for negative trial results.
This study did not intend to elucidate the effectiveness or efficacy of acupuncture. Studies with positive results were identified in order to support the applicability through consideration of comparable adverse effects. The results indicated that of the included trials the reports of positive results were over half (56%) with no serious adverse effects being reported. Therefore, this study indicated that the acupuncture protocols in RCTs are safe and effective.
The three items in the performance aspect of the external validity checklist proved to be important. They were not assessed by reporting but were used for assessing the applicability of trial results. Furthermore, they could help practitioners in terms of clinical decision making. For example, the appropriate control intervention item, one of the STRICTA criteria, influences the overall research design and the trial conclusion. A questionable conclusion could be harmful and lead to the distortion of trial applicability and thus, clinical decisions. The data from this review indicated that most of the included RCTs (87.5%) on acupuncture for knee pain applied the control intervention (s) that matched their research question(s). Another example of performance criterion is the outcome measure applied, which should be a patient-centred outcome or a surrogate outcome.5 ,12 Biological or imaging markers or recorded usage of medication are not yet suitable as primary outcome measures.45 In this study, pain and disability, which were the relevant clinical outcomes as well as the quality of life and the global measurement, were given more weight according to the core set of outcome measures by Outcome Measures in Rheumatoid Arthritis Clinical Trials (OMERACT).45 ,46 The other outcomes, such as number of days admitted to hospital, the physical performance test and amount of medication used, were not used as specific items on our external validity scale.
To study the differences of acupuncture intervention between research and practice, the study included data from different sources of practice. The data on points and treatment parameters provided by 11 acupuncturists who were at that time working in the hospital reflected treatment in real clinical practice. Most of the Thai textbooks (four out of five) provided a treatment protocol associated with the cause of diseases and treatment concepts according to TCM. Otherwise, over half of the English textbooks (14 out of 21) provided the acupoints and treatment parameters representing the Western style of acupuncturists in practice. Despite the small number of the acupuncturists participating and the limitations of the literature, our results could be used to help identify the identity and similarity of the common points and the treatment parameters among the different sources, which could then be considered an alternative protocol of acupuncture for OA knee pain.
Some of the acupoints (26.7%) used by RCTs were not in line with the literature and the surveyed acupuncturists. However, most traditional Chinese acupuncturists tended to make their own diagnoses under the assumption that each patient is unique, irrespective of conventional diagnosis. Therefore, the eight points that were not commonly used (ST6, SP4, GB41, BL20, BL58, BL62, LR3 and TE5) might depend on the clinical condition of the patients in each trial or different techniques of the acupuncturist providing the intervention. Also, SP10, one of the most common points used in RCTs (60.0%) appeared in RCTs and was highly recommended by acupuncturists (72.7%) but was barely mentioned in the literature (5.0%).47 Since SP10 is a common point traditionally used to improve the vessel/nourish the Blood, it was a supplementary point used in combination with other treatments of pain caused by Bi syndrome.48 So one might not expect to find SP10 contained within the specific regimen for knee pain in most of the textbooks used in this review. For the treatment frequency, one or two times per month was not supported by any of the textbooks or acupuncturists included in the study but was found in two RCTs; it could be possible to use such a frequency following the main course of treatment (normally twice per week) in order to maintain the effects of the acupuncture over the long term.
Internal validity seems to be more critical and is a prerequisite for external validity.6 Many funding agencies and journals tended to be more concerned with the scientific rigor of interventions studied than with the applicability of the results.15 Several studies have addressed topics relating to external validity to facilitate the applicability of the trial results.5 ,6 ,8,–,10 ,14 ,15 Striking a balance between external validity (generalisability) and internal validity (reliability of the results) is difficult. For example, standardised protocols are very necessary for clinical research49,–,51 but this is not the nature of TCM in which practitioners prefer to make their own individual diagnoses and treatments. Moreover, the practice of cointervention such as the intake of medication may lead to the distorted results and misinterpretation. However, a protocol that is too stringent would make the trial inappropriate and far different from normal practice. Therefore, giving an explicit report of external validity is necessary and helps gain further understanding of the practical application of the trial results. Though efforts of researchers and editors to improve the reporting of the results of RCTs in acupuncture, such as STRICTA guidelines, have been established, most of them have emphasised the reporting of acupuncture intervention. However, this problem has not been restricted to the field of acupuncture research, as many systematic reviews of healthcare have now confirmed that RCTs do often lack external validity.5 Great efforts should be extended to different aspects of external validity to allow clinicians to judge to whom and in which context these results could reasonably be applied.
One resource limitation was that of the searching of the databases on RCTs for studies in languages other than English. As for the supporting literature of acupuncture treatment, only works in Thai and English were included. Although we used convenience sampling for the survey and only communicated with the acupuncturists who worked in hospitals in the northern, central and eastern parts of Thailand, the results from the RCTs showed to a great extent the similarity of points and treatment parameters with the literature and acupuncturist practice.
The reporting of acupuncture RCTs for knee pain is inadequate. This study highlighted the lack of consideration of other aspects of external validity beyond the STRICTA guidelines in the published reports of RCTs acupuncture for knee pain. Greater attention should be paid to external as well as internal validity to increase the applicability of a trial's results. Attempts to provide detailed information on the acupuncture protocol for knee pain based on the applications in RCTs together with the sources should lead to a more robust protocol for future clinical trials and acupuncturist practice.
We checked whether studies of acupuncture for OA knee are generalisable.
Acupuncture generally represented normal clinical practice.
Study design rarely included quality of life, or satisfaction.
The authors would like to thank the acupuncturists who provided the information to this study. We also thank Chulalongkorn University Centenary Academic Development Project-12, which funded this research.
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Web Only Data - This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Web Only Data - This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Funding Chulalongkorn University.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Additional unpublished data can be provided to the readers via the email by contacting the corresponding author.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.