Article Text

PDF
Assessment of reporting quality in randomised controlled trials of acupuncture for post-stroke rehabilitation using the CONSORT statement and STRICTA guidelines
  1. Jingchun Zeng1,
  2. Guohua Lin2,
  3. Lixia Li3,
  4. Liming Lu4,
  5. Chuyun Chen3,
  6. Lihong Lu1
  1. 1Guangzhou University of Chinese Medicine, Guangzhou, China
  2. 2Department of Acupuncture, First Affiliated Hospital, Guangzhou University of Chinese Medicine, Guangzhou, China
  3. 3Department of Acupuncture, Guangzhou Hospital of Traditional Chinese Medicine, Guangzhou, China
  4. 4Key Unit of Methodology in Clinical Research, Guangdong Provincial Hospital of Chinese Medicine, Guangzhou, China
  1. Correspondence to Professor Guohua Lin, Department of Acupuncture, First Affiliated Hospital, Guangzhou University of Chinese Medicine, Guangzhou 510405, China; tcmlin-801{at}163.com

Abstract

Objectives To evaluate the completeness of reporting of randomised controlled trials (RCTs) of acupuncture for post-stroke rehabilitation in order to provide information to facilitate transparent and more complete reporting of acupuncture RCTs in this field.

Methods Multiple databases were searched from their inception through September 2015. Quality of reporting for included papers was assessed against a subset of criteria adapted from the Consolidated Standards for Reporting Trials (CONSORT) 2010 statement and the Standards for Reporting Interventions in Controlled Trials of Acupuncture (STRICTA) guidelines. Each item was scored 1 if it was reported, or 0 if it was not clearly stated. Descriptive statistical analysis was performed. Cohen's κ-statistics were calculated to assess agreement between the two reviewers.

Results A total of 87 RCTs were included in the full text. Based on CONSORT, good reporting was evident for items ‘‘Randomised’ in the title or abstract’, ‘Participants’, ‘Statistical methods’, ‘Recruitment’, ‘Baseline data’, and ‘Outcomes and estimation’, with positive rates >80%. However, the quality of reporting for the items ‘Trial design’, ‘Outcomes’, ‘Sample size’, ‘Allocation concealment’, ‘Implementation’, ‘Blinding’, ‘Flow chart’, ‘Intent-to-treat analysis’, and ‘Ancillary analyses’ was very poor with positive rates <10%. Based on STRICTA, the items ‘Number of needle insertions per subject per session’, ‘Responses sought’, and ‘Needle type’ had poor reporting with positive rates <50%. Substantial agreement was observed for most items and good agreement was observed for some items.

Conclusions The reporting quality of RCTs in acupuncture for post-stroke rehabilitation is unsatisfactory and needs improvement.

Statistics from Altmetric.com

Introduction

Stroke is the second most common cause of death, as well as the leading cause of adult disability worldwide.1 Due to an ageing population, the prevalence of stroke is expected to rise significantly around the world. In addition to the substantial mortality associated with stroke, post-stroke rehabilitation places a heavy burden on patients' families and society as a whole.2 Moreover, there are still few effective interventions for stroke and stroke-related disorders such as post-stroke motor dysfunction and post-stroke dysphagia.3 In China and elsewhere in East Asia, acupuncture has played a primary role in the treatment of stroke-related disorders over the past 1000 years;4 more than half of local physicians in China and Korea report routinely using acupuncture as an intervention for stroke-related disorders.5–7 More recently, a growing number of physicians and patients in Western countries and emerging economies have begun to accept acupuncture as an alternative therapy for a variety of neurological disorders.8–10

In recent years, studies focused on demonstrating the effectiveness of acupuncture for stroke-related disorders have drawn contradictory conclusions. Some reviews have demonstrated that acupuncture treatment is superior to either no acupuncture or conventional therapy for stroke.1 ,3 Meanwhile, other reviews have suggested that too little evidence exists to prove the efficacy or effectiveness of acupuncture for stroke.11 ,12 Researchers have questioned the reporting quality of the randomised controlled trials (RCTs) included in these reviews. It is important to evaluate the quality of reporting in RCTs of acupuncture for stroke rehabilitation; this is because high quality reporting is essential to guide scholarly peer-review decisions and to conduct unbiased meta-analysis, as it influences our interpretation of the evidence. The Consolidated Standards for Reporting Trials (CONSORT) statement13 and the acupuncture-specific Standards for Reporting Interventions in Controlled Trials of Acupuncture (STRICTA) guidelines14 were designed to help authors fully report details of RCTs and acupuncture interventions, respectively.

The aim of this study was to evaluate the completeness of reporting of RCTs of acupuncture for post-stroke rehabilitation based on the CONSORT statement and STRICTA guidelines. To the best of our knowledge, no study has previously investigated the quality of reporting of trials in this particular area against these particular standards. We also aimed to provide the information necessary to make recommendations to facilitate transparent and more complete reporting of acupuncture RCTs in this field.

Methods

Search strategy

The following databases were searched from their inception through September 2015: the Cochrane Central Register of Controlled Trials (CENTRAL), Ovid Medline, Ovid Embase, Ovid Allied and Complementary Medicine (AMED), the Chinese Biomedical Database (CBM), China National Knowledge Infrastructure (CNKI), Wanfang Database and Chongqing Weipu (VIP). Other resources included reference lists of eligible studies and previous systematic reviews that were also reviewed to identify further eligible studies. The following combination of search terms was used in Chinese and English: (acupuncture OR electro-acupuncture OR auricular acupuncture OR scalp acupuncture OR needle OR acupuncture point OR meridian OR acupoint OR jingluo OR Zhenjiu OR Zhenci OR Dianzhen) AND (cerebrovascular disorders OR basal ganglia cerebrovascular disease OR brain ischaemia OR carotid artery diseases OR intracranial arterial diseases OR intracranial arteriovenous malformations OR intracranial embolism OR intracranial haemorrhages OR stroke OR brain infarction). All articles were published either in Chinese or in English. The details of the search strategy can be found in online supplementary file 1.

Inclusion and exclusion criteria

Types of studies

We included any studies with ‘RCT’ mentioned in the title, abstract or text that evaluated the efficacy or effectiveness of acupuncture to treat any type of post-stroke-related disorder including post-stroke hemiplegia, dysphagia, depression, hiccups, urinary incontinence, insomnia, hand-shoulder syndrome, and shoulder pain, regardless of whether allocation concealment or blinding were used or not. RCTs involving any acupuncture modality were included. RCT reports also involving the use of other alternative therapies or drug therapy were included provided the same non-acupuncture therapies were given to both the intervention (acupuncture) arm and the control (no acupuncture) arm. Reports were excluded if they: (1) were abstracts only or not published in full; or (2) compared the efficacy or effectiveness of different acupuncture therapies, such that there was no non-acupuncture control group.

Types of participants

Patients of any gender, sex or ethnicity with ischaemic or haemorrhagic stroke in the subacute or chronic phases were eligible. The stroke had to have been diagnosed according to the WHO's definition—rapidly developing clinical signs of focal (or global) disturbances of cerebral function lasting >24 hours or leading to death, with no other apparent cause other than one of vascular origin15 and/or confirmed by MRI or CT.

Types of interventions

The study group included any type of invasive acupuncture including manual acupuncture (MA), electroacupuncture (EA) or auricular (ear) acupuncture. Studies using different forms of needle stimulation and different needling techniques were also included, but studies comparing non-invasive techniques such as laser acupuncture or acupressure were excluded. The control groups included the use of placebo, treatment as usual, no treatment, or other active interventions. Moxibustion trials were not included in our study.

Selection of reports

Firstly, one researcher (JZ) picked out duplicate reports and scanned the title and abstract of the citations retrieved by the selection search engine using NoteExpress. Two researchers (JZ and LL) then viewed the full text of all potentially eligible reports obtained separately. If an article did not meet the inclusion criteria and/or met one or more exclusion criteria (eg, non-controlled trials, animal experiments, reviews, comments, inappropriate control group, other diseases or stages) then it was moved into the appropriate folder with labels in NoteExpress. Several controversial articles were labelled as ‘suspicious’, in which case a final decision was deferred until the next selection stage. Any disagreements between the two researchers were discussed by the whole team and a consensus was reached (figure 1).

Figure 1

Flow chart illustrating selection of randomised controlled trials (RCTs) of acupuncture for post-stroke rehabilitation.

Data extraction

Two researchers (JZ and LL) extracted general information and performed the assessment with respect to CONSORT and STRICTA standards on the final included reports using EpiData software (EpiData Association, Denmark), available at http://www.epidata.dk. Data extracted included report title, journal name, year of publication, journal type, publication language, and whether the research was funded or not. Any missing information was coded as ‘not reported’ and disagreements were resolved after obtaining more information from related references (published protocols and/or previous methods papers referred to in the included reports) before finalising the data extraction. Two reviewers scanned the full text of all 87 reports and finished this step.

Assessment of reporting quality

We assessed the quality of the included reports according to the CONSORT and STRICTA standards. The CONSORT statement comprises items including abstract, background, participants, interventions, objectives, outcome measures, sample size, sequence generation, allocation concealment, implementation, blinding, statistical methods, participant flow, recruitment, demographic data, numbers analysed, outcomes and estimation, ancillary analyses, and harm. The CONSORT discussion section items were excluded because we considered them too subjective to evaluate. For the ‘interventions’ item, we used STRICTA to evaluate the reporting quality instead of CONSORT, as STRICTA has more detailed and specific standards for reporting acupuncture interventions.

STRICTA consists of items including acupuncture points used, number of needles inserted, depths of insertion, response elicited, needle stimulation, needle retention time, needle type, practitioner background, duration of training, length of clinical experience, and expertise. All of this information can be found in tables 1 and 2.

Table 1

Reporting quality using items from the CONSORT statement (n=87 studies)

Table 2

Reporting quality of details of needling from STRICTA (n=87)

Each article was reviewed by two independent investigators (JZ and LL), who had been trained in research methodology and statistics, using the CONSORT and STRICTA checklists. Each item was scored 1 if it was reported and 0 if it was not clearly stated. After both raters finished their review, the authors calculated Cohen's κ-statistics to assess agreement between the two reviewers. Agreement was judged as poor if κ was ≤0.2, fair if 0.2 <κ ≤0.4, moderate if 0.4 <κ ≤0.6, substantial if 0.6< κ ≤0.8, good if κ was >0.8, and perfect if κ=1. Any difficulties or disagreements in the process were solved through discussion. The number, percentage, and 95% CIs of each variable were summarised using the Statistical Package for the Social Sciences (SPSS) V.20.0 (SPSS Inc, Chicago, Illinois, USA). Cohen's κ-statistics were performed using SAS V.9.3 (SAS Institute Inc, North Carolina, USA).

Results

Characteristics of included trials

Year of publication

A total of 87 full-text RCTs were included. When counting the number of articles, it was noted that the frequency of publication of RCTs of acupuncture for post-stroke rehabilitation had increased over time (figure 2).

Figure 2

Number of randomised controlled trials of acupuncture for post-stroke rehabilitation published per calendar year between 1997 and 2015.

Journals and languages

The 87 papers were published in 58 different journals. Eight papers (9.2%) were published in The Shanghai Journal of Acupuncture and Moxibustion, five (5.7%) in Chinese Acupuncture and Moxibustion, four (4.6%) in The Journal of Clinical Acupuncture, and three (3.4%) in The World Journal of Acupuncture-Moxibustion. Of the 87 RCT papers, 79 (90.8%) were written in Chinese. The remaining eight papers (9.2%) were published in English and appeared in the following six journals: The World Journal of Acupuncture-Moxibustion, The Journal of Musculoskeletal Pain, Clinical Rehabilitation, The Journal of the American Physical Therapy Association, Alternative Therapies, and The Journal of Alternative and Complementary Medicine. Regarding journal category, 24 (27.6%) were published in general medical journals, 11 (12.6%) in specialist medical journals, and 52 (59.8%) in traditional or alternative medicine journals.

Funding sources

Only 16 articles (18.4%) reported their sources of funding. Funding was obtained from national, provincial/municipal and regional sources in six (6.9%), eight (9.2%), and two cases (2.3%), respectively. Funding from international pharmaceutical companies or research agencies was not reported in our sample.

Quality of reporting

Reporting quality based on CONSORT

The ratings of reporting quality based on the CONSORT statement are listed in table 1. Good reporting was evident for items ‘‘Randomised’ in the title or abstract’, ‘Participants’, ‘Statistical methods’, ‘Recruitment’, ‘Baseline data’, and ‘Outcomes and estimation’, with positive rates >80%. However, the quality of reporting in items ‘Trial design’, ‘Outcomes’, ‘Sample size’, ‘Allocation concealment’, ‘Implementation’, ‘Blinding’ ‘Flow chart’, ‘Intent-to-treat analysis’, and ‘Ancillary analyses’ was very poor with positive rates <10%. Substantial agreement was observed for items 2, 6, 9, 10, 12, 13, 14, 15, 16, 17, and 18. Good agreement was observed for items 3, 4, 7, 8, 11, and 19. Perfect agreement was observed for item 1.

Reporting quality based on STRICTA

The reporting quality ratings for the needling details based on the STRICTA guidelines are listed in table 2. Good reporting existed for the items ‘Names of points used’, ‘Needle stimulation’, and ‘Needle retention time’, with positive ratings >80%. However, the reporting quality for items ‘Number of needle insertions per subject per session’, ‘Responses sought’, and ‘Needle type’ was poor, with positive rates <50%.

Substantial agreement was observed for the items ‘Number of needle insertions’, ‘Responses sought’, ‘Needle stimulation’, and ‘Needle retention time’. Good agreement was observed for items ‘Names’, ‘Depth of insertion’, and ‘Needle type’.

Discussion

This study has demonstrated that the reporting quality in RCTs of acupuncture for post-stroke rehabilitation between 1997 and 2015 was unsatisfactory. Many of the RCTs did not adhere to CONSORT and STRICTA guidelines. It is necessary for us to pay attention to this issue. Firstly, well-conducted RCT reports benefit other researchers by helping to evaluate the internal validity of the results and making it easier to replicate research. High quality reporting also supports the performance of less biased systematic reviews and aids medical decision-making. Secondly, with accurate and standardised reporting, acupuncturists can more easily master effective operational processes following the guidance of RCT reports.

For some items, information was insufficient or inadequate in most studies. These items were ‘Trial design’, ‘Outcomes’, ‘Sample size’, ‘Allocation concealment’, ‘Implementation’, ‘Blinding’, ‘Flow chart’, ‘Intent-to-treat analysis’, and ‘Ancillary analyses’ from the CONSORT statement, and ‘Number of needle insertions per subject per session’, ‘Responses sought’, and ‘Needle type’ from the STRICTA guidelines.

The common limitations in the above items illustrate that the reporting in these trials was unreliable. For example, three key methodological considerations—concealment of allocation, appropriate blinding, and intent-to-treat analysis—are crucial for avoiding bias and subsequent distortion of the estimation of the effect.16 Indeed, an overestimation of treatment effects has been demonstrated in trials that were inadequately blinded or in which randomisation was inadequately concealed in comparison with trials that adequately reported these two methodological items.17 Sample size calculation also needs to be reported as it enables readers to know whether or not the trial has enough subjects to detect clinically significant differences between intervention and control groups. Presentation of the relevant information in a flow chart, including the number of participants who did not receive the intervention as allocated or did not complete treatment, permits other researchers to intuitively master the trial process and evaluate a trial's validity. As for STRICTA, details of the acupuncture intervention need to be clearly reported. This benefits other researchers seeking to replicate the acupuncture intervention, to evaluate the internal validity of the clinical results, and to promote the application of an effective intervention in clinical practice. This is especially important for acupuncture, as an empirical medicine, because different acupuncturists providing treatment to different study arms may influence the generalisability of the trial results. Therefore the details of the intervention need to be explained using a structured format.

This study has several limitations. Firstly, we only reviewed literature published in English and Chinese. As acupuncture research is also flourishing in South Korea and Japan, it might be necessary to include RCTs published in Korean or Japanese in this field to perform a more comprehensive and confident review. Secondly, in order to evaluate the quality of reporting in RCTs using a quantitative method and taking into account the purpose of our study, we only extracted major items (rather than all items) from the 2010 CONSORT statement and the STRICTA guidelines.

In conclusion, we have demonstrated that reporting quality of RCTs of acupuncture for post-stroke rehabilitation is unsatisfactory and needs improvement. Thus, we suggest that authors adhere to the 2010 CONSORT statement and the STRICTA guidelines when reporting RCTs in this field in the future. Furthermore, we suggest that journal editors and their editorial offices should more strongly promote these guidelines, and use them to assist authors during pre-submission enquiries and when revising submitted manuscripts. This is particularly pertinent since it has been found that the majority of editors of Chinese medical journals are not familiar with the content of the CONSORT statement and its extension papers.18 Finally, external peer reviewers should also consider these guidelines when judging manuscripts in order to further improve the reporting quality of acupuncture RCTs.

References

View Abstract

Footnotes

  • Contributors Study concepts: Jingchun Zeng. Study design: Guohua Lin and Lixia Li. Data acquisition: Liming Lu, Chuyun Chen and Lihong Lu. Quality control of data and algorithms: Jingchun Zeng, Liming Lu and Lixia Li. Data analysis and interpretation: Jingchun Zeng and Guohua Lin. Statistical analysis: Jingchun Zeng and Lixia Li. Manuscript preparation: Jingchun Zeng, Guohua Lin and Lixia Li. Manuscript editing: Liming Lu, Chuyun Chen and Lihong Lu. Manuscript review: Guohua Lin and Lixia Li.

  • Funding This research was supported by the Guangzhou Science and Technology Planning Project (contract no. 201508020002 and approval no. (2015) 134), the National Natural Science Foundation of China (grant no. 81574061), and the Guangdong Natural Science Foundation (project no. 2016A030310290).

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.