Objectives To investigate the quality of reporting for randomised controlled trials of acupuncture for neurological disorders conducted in China before and after the implementation of the Consolidated Standards of Reporting Trials (CONSORT) and Standards for Reporting Interventions in Controlled Trials of Acupuncture (STRICTA) guidelines.
Methods The quality of reporting for included papers was assessed against a subset of criteria adapted from CONSORT and STRICTA. CONSORT and STRICTA were developed in 1996 and 2001, respectively. Thus, for the date of publication we selected 2-year periods, at 5-yearly intervals: 1994–1995; 1999–2000; 2004–2005 and 2009–2010. These selections cover the periods before the publication dates of both guidelines (1996, 2001) and at least 3 years afterwards, and provide reasonably up-to-date data. We calculated the total score for each guideline and compared reported differences during different date ranges.
Results For CONSORT items (maximum score 8), there was evidence of a slight improvement in reporting between 1994–1995 and 1999–2000 combined (2.5±0.6) and 2004–2005 and 2009–2010 combined (3.0±0.9) (difference 0.5, 95% CI 0.2 to 0.8). For STRICTA items (maximum score 17), there was evidence of a slight improvement in reporting between 1994–1995 and 1999–2000 combined (8.9±1.8) and 2004–2005 and 2009–2010 combined (10.3±1.6) (difference 1.4, 95% CI 0.9 to 1.9).
Conclusions The quality of reporting for studies of acupuncture for neurological disorders has generally improved since the implementation of STRICTA and CONSORT guidelines.
Statistics from Altmetric.com
Neurological disorders (ND), referring to any disorder of the nervous system,1 encompass a group of clinically heterogeneous diseases, some of which are commonly associated with motor impairment syndromes. They are an important cause of mortality and constitute 12% of total deaths globally. They represented 11%, 7%, 9% and 5% of the global burden of disease in high, upper middle, lower middle and low income countries, respectively, in 2005.2 This is a greater burden than digestive diseases, respiratory diseases and malignant neoplasms. Acupuncture is mainly used to improve motor, sensation, speech and other neurological functions in patients with ND. Compared with other conventional interventions, acupuncture is relatively simple, inexpensive and safe, and has been well accepted by Chinese patients and is also increasingly practised in some Western countries. Recently, several randomised controlled trials (RCTs) on acupuncture for ND have been conducted in China, but as yet there has been no systematic evaluation of reporting quality of RCTs in this field. Quality of reporting is essential for guiding journal peer-review decisions and experts’ recommendations, conducting unbiased meta-analysis and influencing our interpretation of evidence.3 Therefore it is important to evaluate the reporting quality status of RCTs related to acupuncture for ND conducted in China.
Two initiatives have aimed at improving reporting standards. The Consolidated Standards of Reporting Trials (CONSORT) statement is an international consensus expert guideline developed in 1996, revised 5 years later and last updated in 2010.4 The acupuncture-specific Standards for Reporting Interventions in Controlled Trials of Acupuncture (STRICTA) were compiled and published in late 2001/early 2002,5–7 and aimed at improving the reporting of acupuncture interventions. The combination of both tools helped us to evaluate the completeness and transparency of RCT reports on acupuncture for ND according to the CONSORT and STRICTA items.
Prady et al8 evaluated the impact of STRICTA and CONSORT on quality of reporting for acupuncture trials. Their research included only articles published in English and its scope was too broad to discern the reporting quality status of RCTs in acupuncture for ND. To the best of our knowledge, no data for the quality of RCT reporting in acupuncture for ND have been obtained in China. This study is therefore designed to assess the reporting quality of RCTs related to acupuncture for ND conducted in China and the influence of the introduction of STRICTA and CONSORT on this reporting.
We searched the Trials Register of the Cochrane Complementary Medicine Field (February 2014) and the Chinese Acupuncture Trials Register (February 2014). In addition, we searched the following bibliographic databases: Medline (1966 to February 2014), Embase (1980 to February 2014), AMED (the Allied and Complementary Medicine Database, 1985 to February 2014), CINAHL (1982 to February 2014), the Chinese Biological Medicine Database (CBM-disc, 1979 to February 2014), China National Knowledge Infrastructure (CNKI, 1979 to February 2014), VIP database (1992 to February 2014) and Wanfang databases (1982 to February 2014). The following search terms were used in Chinese and English: parkinson, multiple sclerosis, cerebral palsy, alzheimer, ischaemic stroke, facial paralysis, shingles, hemiplegia, neuropathic pain, nerve deafness, vascular dementia, nerve palsy, neuritis, sciatica, lumbar disc herniation, nerve damage, insomnia, epilepsy, huntington, dementia, ictus, syncope, amyotrophic lateral sclerosis, maladie de charcot, lou gehrig disease, muscular dystrophy, brain tumour, meningitis, poliomyelitis, neurological, neuromuscular, acupuncture, electro-acupuncture, needle, scalp acupuncture, auricular acupuncture, acupuncture point, meridian, acupoint, point injection, China, Chinese, randomised controlled trial, etc.
We aimed to establish a baseline for reporting quality, and track changes over time after publication of CONSORT and STRICTA. Previous studies have argued that a period of 3–4 years after the publication of reporting standards is sufficient to ensure promotion and adoption of the new guideline.8 ,9 CONSORT and STRICTA were developed in 1996 and 2001, respectively. Thus, for the date of publication we selected 2-year periods, at 5-yearly intervals: 1994–1995; 1999–2000; 2004–2005; and 2009–2010. These selections cover the periods before the publication dates of both guidelines (1996, 2001) and at least 3 years afterwards, and provide reasonably up-to-date data. Databases continue to accumulate studies for some years, but it is reasonable to regard the 2010 databases as complete. We did not include the publication date of 2013–2014 as the databases are incomplete.
Inclusion and exclusion criteria
Types of studies
RCTs comparing acupuncture with at least one control group that used placebo, sham treatment or conventional treatment in patients with ND were included. RCTs were included if invasive acupuncture was used as the sole intervention or as an adjunct to another standard treatment for ND and if the control group received the same concomitant treatments as the acupuncture group. A randomisation sequence generated from a random number table, calculator or computerised random number generator was considered acceptable. All RCTs had to be performed in China and mainly by Chinese researchers.
Types of interventions
Acupuncture therapy must be invasive and defined as body acupuncture, scalp acupuncture, electroacupuncture, warm needling or ear acupuncture, etc. We use the term placebo acupuncture to refer to a needle attached to the skin surface (not penetrating the skin). Sham acupuncture refers to (1) a needle placed in an area close to, but not in, the acupuncture points; (2) subliminal skin electrostimulation via electrodes attached to the skin.
Assessment of quality of reporting
CONSORT assessment checklist
We chose eight items from five categories of CONSORT to understand the change of reporting quality of acupuncture RCTs for ND over time. Randomisation, concealment of allocation, appropriate blinding (either practitioner, participant or assessor), baseline characteristics and analysis according to the intention-to-treat (ITT) principle were items chosen by previous evaluation studies of CONSORT10–12 as they relate to potential sources of systematic bias13 and are present in both the original and revised versions of the checklist.
STRICTA assessment checklist
We converted the STRICTA guidelines into a reporting quality assessment checklist with 17 items, including acupuncture rationale, points used, number of needle insertion, depths of insertion, response elicited, needle stimulation, needle retention time, needle type, treatment regimen, co-interventions, practitioner background, duration of training, length of clinical experience, expertise and control interventions, etc.
An overall quality score with eight items from CONSORT and 17 items from STRICTA were used (tables 2 and 3). Each item was scored 1 if it was reported and 0 if it was not clearly reported, or definitely not stated. The score ranges of CONSORT and STRICTA were 8 and 17, respectively.
Data extraction and analysis
Each article was reviewed by two independent investigators who studied the CONSORT and STRICTA standards before making an assessment to ensure correct understanding of every item listed in the two checklists. If they had any difficulties or disagreements in the process, these were resolved by discussion. Included reports were then assessed according to the two standards respectively.
Data were summarised for each time period. We calculated the proportion of articles reporting each item of CONSORT and STRICTA during different publication dates of 1994–1995, 1999–2000, 2004–2005 and 2009–2010. We compared the proportions of reporting of each item between different date ranges by χ2 test. In addition, we calculated the total score for reporting each guideline. After combining studies into two groups, differences were compared by t test. Differences among three or more subgroups were compared by analysis of variance. Descriptive statistical analysis and statistical inference were performed using SPSS V.20.0 (SPSS, Illinois, USA).
The RCT selection process is outlined in figure 1. After identification, screening, determination of the eligibility of the reports, a total of 226 relevant RCTs were included in the final analysis. A list of the included RCTs is provided in the online supplementary data.
Characteristics of the included trials
The frequency of RCTs of acupuncture for ND was found to change over time: 13 in 1994–1995, 36 in 1999–2000, 134 in 2004–2005, and 43 in 2009–2010. Of the 226 RCT papers, 61 (27%) were published in general medical journals, 122 (54%) in specialist medical journals and 43 (19%) in journals focusing on complementary and alternative medicine. Thirty-eight articles (16.8%) reported their sources of funding. Funding was obtained from provincial/municipal or international institutes. Control interventions of 145 trials (64.2%) comprised drugs, followed by sham acupuncture (78 reports; 34.5%), care as usual/waiting list/no intervention (one report; 0.4%) and non-penetrating sham needles (two reports; 0.9%).
Quality of reporting
Variation in CONSORT reporting over time
Table 1 presents the reporting of items before and after the introduction of CONSORT. The least reported items (reported ≤5% of the time) were sections B (allocation concealment) and C (blinding) in most years. The most frequently reported items (reported ≥70% of the time) were section A (randomisation sequence) and E (ITT analysis). There was evidence that most items (except E, ITT analysis) were reported more frequently after the implementation of CONSORT (p<0.05 or p0.01). Figure 2 displays the percentages of actual reports in 1994–1995, 1999–2000, 2004–2005 and 2009–2010, which for CONSORT are 25.0%, 33.3%, 34.8% and 45.6%, respectively; and for STRICTA are 51.6%, 52.5%, 59.6% and 62.9%, respectively.
Table 2 illustrates the improved mean scores of CONSORT items reported over time. There was evidence of a slight improvement in reporting between 1994–1995 and 1999–2000 combined (2.5±0.6) and 2004–2005 and 2009–2010 combined (3.0±0.9) (difference 0.5, 95% CI 0.2 to 0.8).
Variation in STRICTA reporting over time
Table 3 presented the reporting of items before and after the introduction of STRICTA. The least reported items (reported ≤15% of the time) were items 4b (setting and context of treatment), 5 (description of participating acupuncturists) and 6a (rationale for the control or comparator). The most frequently reported items (reported ≥70% of the time) were items 1a (style of acupuncture), 2a (number of needle insertions), 2b (names of points used), 2e (needle stimulation), 2f (needle retention time), 3a (number of treatment sessions), 3b (frequency and duration of treatment sessions) and 6b (precise description of the control or comparator). There was evidence that three items (2c, depth of insertion; 2f, needle retention time; 2g, needle type) were reported more frequently after the implementation of STRICTA (χ2=6.0, p<0.05; χ2=11.6, p<0.01; χ2=7.0, p<0.01; χ2=11.2, p<0.01; χ2=6.0, p<0.05). The percentages of reports for STRICTA before and after 2001 are 52.2% and 66.1%.
Table 4 illustrates the improvement in mean scores for reporting STRICTA items over time. There was evidence for a slight improvement in reporting between 1994–1995 and 1999–2000 combined (8.9±1.8) and 2004–2005 and 2009–2010 combined (10.3±1.6) (difference 1.4, 95% CI 0.9 to 1.9, p<0.05).
To the best of our knowledge, this study is the first investigation into the quality of reporting on RCTs of acupuncture for ND according to the revised CONSORT and STRICTA guidelines. Our study found that overall, the quality of reporting improved for studies published after 1996 and 2001 compared with those published before 1996. The trend is for improving quality of reporting over time. The total scores of CONSORT and STRICTA were not high owing to the limited proportion of individual items reported.
The purpose of reporting guidelines such as CONSORT and STRICTA is to increase the transparency of study methods and ultimately improve the overall quality of research. Poor reporting of trials make it difficult for readers to assess the validity of the results and may mislead policy-makers in their decisions. We found evidence that the introduction of the two guidelines improved the reporting quality to some extent. In a previous study, Prady et al8 concluded that reporting of CONSORT items had improved after the introduction of CONSORT but that the introduction of STRICTA did not improve the reporting of STRICTA items. The conclusions were limited by the small number of reports.8 Our study has greater power as the sample size is larger. An evaluation of auriculotherapy trials came to conclusions similar to ours.14 Other studies have argued that CONSORT and STRICTA would improve the standardisation of the reporting of RCTs of acupuncture.15 ,16
We identified two sections of CONSORT where information was insufficient or inadequate in most studies, leading to a lower total reporting score—allocation concealment and blinding. Others have found the same,8 ,17–19 showing a suboptimal reporting quality. Allocation concealment and blinding are critical in avoiding selection, performance/detection and attrition bias, respectively. An overestimation of treatment effects has been demonstrated in trials with inadequate key methodological design in comparison with trials that adequately reported these methodological items.20
For STRICTA, ‘setting and context of treatment’, ‘description of participating acupuncturists’ and ‘rationale for the control or comparator’ were the least reported items. ‘Setting and context of treatment’ could have important additional treatment effects.21 ‘Description of participating acupuncturists’ referring to the eligibility criteria for acupuncturists should also be explained. Additionally, ‘rationale for the control or comparator’ should be presented and justified in relation to the research question and the methodology.
A possible shortcoming is that we sampled RCTs of acupuncture for ND regardless of whether the publishing journal had adopted STRICTA or CONSORT. Most of the included reports were published in Chinese journals, which did not clearly state whether they had adopted STRICTA or CONSORT on their websites or in printed publications. Another limitation is that owing to a language barrier, we did not search for any manuscripts published in languages other than Chinese or English. This is a worthwhile area for future study. It remains unknown whether searching in other language journals would have altered the constitution of our sample or the results.
This research provides a formal assessment of the quality and reporting of RCTs on acupuncture for ND conducted in China. We expect it to contribute to the gradual improvement of this type of trial. The quality of reporting of such trials has generally improved since the implementation of STRICTA and CONSORT guidelines. However, more attention should be paid to reporting ‘allocation concealment’, ‘blinding’, ‘setting and context of treatment’, ‘description of participating acupuncturists’ and ‘rationale for the control or comparator’.
CONSORT and STRICTA guidelines are aimed at improving the reporting of clinical trials.
We assessed whether 226 reports of Chinese trials of acupuncture in nervous diseases improved after publication of these guidelines.
There was some improvement, but room for more.
Review history and Supplementary material
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online supplement
Contribution Study concepts: GL and LL. Study design: GL, LL and JZ. Data acquisition: GL and LL. Quality control of data and algorithms: GL, LL and JZ. Data analysis and interpretation: GL and LL. Statistical analysis: LL and JZ. Manuscript preparation: GL and LL. Manuscript editing: GL and JZ. Manuscript review: GL and LL.
Competing interests None.
Patient consent Obtained.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.