Article Text

PDF

Heterogeneity in search strategies among Cochrane acupuncture reviews: is there room for improvement?
  1. Steve Lui1,
  2. Erica J Smith2,
  3. Mishka Terplan3
  1. 1Department of Nursing and Health Studies, Centre for Health and Social Care, University of Huddersfield, Queensgate, Huddersfield, West Yorkshire, UK
  2. 2Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, Illinois, USA
  3. 3Department of Obstetrics, Gynecology and Reproductive Sciences, The University of Maryland School of Medicine, Baltimore, Maryland, USA
  1. Correspondence to Mr Steve Lui, Department of Nursing and Health Studies, Centre for Health and Social Care, University of Huddersfield, Queensgate, Huddersfield, West Yorkshire HD1 3DH, UK; s.lui{at}hud.ac.uk

Abstract

Objective Given the international focus and rigorous literature searches employed in Cochrane systematic reviews, this study was undertaken to evaluate strategies employed in Cochrane reviews and protocols assessing acupuncture as a primary or secondary intervention.

Methods The Cochrane Collaboration of systematic reviews was searched in February 2009 for all reviews and protocols including information on acupuncture. Information was abstracted from all retrieved articles on review status, type and number of English and Chinese language databases searched, participation of at least one Chinese speaking author and language restriction. Frequencies were calculated and bivariate analyses were performed stratifying on interventions of interest to assess differences in search strategy techniques, language restrictions and results.

Results The search retrieved 68 titles, including 48 completed reviews, 17 protocols and three previously withdrawn titles. Acupuncture was the primary intervention of interest in 44/65 (67.7%) of the retrieved reviews and protocols. While all articles searched at least one English language database, only 26/65 (40.0%) articles searched Chinese language databases. Significantly more articles where acupuncture was the primary intervention of interest searched Chinese language databases (53% vs 9%, p<0.01). Inconclusive findings as to the effectiveness of acupuncture were found in 28/48 (58.3%) of all completed reviews; this type of finding was more common in reviews which did not search any Chinese language databases.

Conclusions It is important for reviews assessing the effectiveness of acupuncture to search Chinese language databases. The Cochrane Collaboration should develop specific criteria for Chinese language search strategies to ensure the continued publication of high-quality reviews.

Statistics from Altmetric.com

Introduction

Acupuncture consists of the application of needles (as well as pressure and heat) to certain points on the body. Although it has been part of traditional Chinese medicine for centuries,1 it has only recently entered into practice in the West. English language clinical descriptions of acupuncture first appeared in Medline in the 1950s. The first randomised trials on acupuncture were not published until the 1970s with the volume of scientific studies increasing since (figure 1).

Figure 1

English language articles found in Medline searching MESH keywords: acupuncture; acupuncture, analgesia; acupuncture, ear; acupuncture points; acupuncture therapy.

The Cochrane Collaboration is an international organisation whose primary aim is to assist healthcare providers, consumers, researchers and policy makers in navigating the often unmanageable amounts of healthcare information by synthesising the evidence within systematic review frameworks. At its core is the collection of Cochrane reviews, a database that contain high-quality, independent evidence to inform healthcare decision-making.2 Cochrane reviews are thought to represent the highest level of evidence on which to base clinical treatment decisions because they are ‘doubly peer reviewed’ and also because of the exhaustive and transparent search criteria employed for reviews.3

The objective of this study was to systematically evaluate strategies employed by Cochrane authors in reviews containing acupuncture as an intervention. Given Cochrane's international focus and the rigorous literature searches employed in the generation of its systematic reviews, we were particularly interested in whether authors had consulted Chinese language databases or had authors fluent in Chinese.

Methods

The Cochrane Collaboration website (http://www.cochrane.org) was searched in February 2009 using the search term ‘acupuncture’ to identify all publications relating to, and containing, any trials on acupuncture.

A data extraction database was developed to capture key methodological and author information for each of the reviews and protocols identified in the search. Key factors of review status (protocol vs completed review), type and number of English and Chinese databases searched, participation of at least one Chinese speaking author, stated translation of foreign language abstracts and language restriction were abstracted. An article was determined to have a Chinese speaking author if it had at least one author with residence in Chinese speaking country or if the review stated that one author reads Chinese. A review was determined to have a language restriction if the authors stated that there was a language restriction or (in the absence of a stated language restriction) there were no non-English articles included in the literature search. Additionally, information as to whether acupuncture was the primary or secondary intervention as well as the outcome and review findings were abstracted.

Frequencies and proportions were calculated to determine method characteristics of retrieved reviews and protocols. Bivariate analyses were performed stratifying on interventions of interest to assess differences in search strategy techniques and language restrictions. Stratification on the use of Chinese language databases in the search strategy was used to assess differences in acupuncture effectiveness findings. When appropriate, χ2 tests with a two-sided p value of <0.05 were used to determine statistical significance between proportions in group. All analyses were performed by using STATA 10.0 (College Station, Texas, USA).

Results

The key word search of the Cochrane website retrieved 68 publications relating to acupuncture, three of which had been previously withdrawn.4,,6 Therefore the analysis includes 65 Cochrane publications: 48 completed reviews7,,54 and 17 protocols 55,,70 (figure 2). Acupuncture was the primary intervention of interest in nearly two-thirds (n=45) of the retrieved reviews and protocols.

Figure 2

Flow chart of included Cochrane reviews.

The number of reviews and protocols assessing the effectiveness of acupuncture on various diseases and conditions has increased in the past 10 years, hitting a peak in 2006–7, when 19 reviews and protocols were published. (table 1) The number of reviews including trials with acupuncture as a secondary intervention has also increased over the past 10 years, with nine reviews including trials using acupuncture in 2008–9.

Table 1

Publication years for included Cochrane reviews and protocols by intervention of interest

All publications retrieved, searched or planned to search English language databases. The majority of these reviews and protocols searched Medline (98.5%), Embase (95.4%) and CENTRAL (92.3%). Only 26 (40%) of the included reviews and protocols searched Chinese language databases. The choice of Chinese databases was not as consistent as that of English language databases. Among reviews and protocols which used Chinese databases, the China Biological Medicine database was the most searched and was used in 20 (76.9%) of included reviews and protocols. This was followed by the Traditional Chinese Medical Literature Analysis and Retrieval System and Chinese National Knowledge Infrastructure databases each of which were used in 10 (15.4%) of the reviews and protocols. Twenty-seven (41.5%) of the included articles had at least one Chinese speaking author. Two-thirds of the reviews and protocols (n=40) had the capacity or stated that they translated Chinese language abstracts to determine inclusion for the review. However, seven (10.8%) articles had a language restriction on the inclusion criteria and only reviewed English language articles (table 2).

Table 2

Search and author characteristics of retrieved reviews and protocols

When acupuncture was the primary intervention of interest, searching Chinese databases was more likely to be included in the search strategy (53% vs 9%, p<0.01). Having a Chinese-speaking author and translating Chinese language abstracts were also more commonly found in reviews where acupuncture was the primary intervention (p<0.01). Interestingly five (11%) reviews in which acupuncture was the primary intervention employed a language restriction (table 3).

Table 3

Comparison of characteristics of retrieved reviews and protocol by primary intervention of interest

The majority of the 48 full reviews (n=28),were inconclusive because of a lack of included studies to fully answer the research question. After stratifying on the use of Chinese databases, a greater proportion of the reviews which did not search Chinese databases were inconclusive as to the effectiveness of acupuncture (68.6% compared with 30.8%).Seven (14.6%) reviews found acupuncture to have a positive effect on the disease of interest, two (15.4%) reviews which searched Chinese databases and five (14.3%) which did not. However, 11 (22.9%) reviews found acupuncture to be not as effective as another standard treatment; this finding was more common in reviews which searched Chinese language databases than in those which did not (46.2% vs 14.3%, respectively) (table 4).

Table 4

Acupuncture effectiveness finings in Cochrane systematic reviews

Discussion

Since its arrival in the USA in the 1970s, acupuncture has gained in popularity and acceptance. In recent years greater numbers of trials have been performed investigating acupuncture for a wide variety of conditions.71 Interventions of interest in the Cochrane reviews retrieved for this study ranged from relief of headaches to muscle pain to substance abuse dependence. As scientific studies have accumulated, researchers have sought to synthesise the data from within the framework of the Cochrane Collaboration. In order to maximise the acceptability and decrease heterogeneity of these reviews for the wide range of conditions, standardised literature search guidelines should be developed.

Acupuncture differs from other healthcare interventions in that it has an extensive history of research and practice outside of the West. Thus it might be argued that the incorporation of Chinese literature especially is key to assessing its effectiveness. However, we observed a large heterogeneity in the databases searched and the incorporation of Chinese language expertise in Cochrane acupuncture reviews. Nearly half of the reviews specifically assessing acupuncture as the primary intervention did not search Chinese language databases. This is concerning as the neglect of Chinese language literature introduces a bias into the systematic reviews as shown by the observation that reviews that did not include Chinese databases were more likely to be inconclusive. Inconclusive reviews may occur because of conflicting data in the literature or because not enough articles are retrieved during the literature search to reach conclusive conclusions. In the case of acupuncture reviews, not searching Chinese databases may substantially limit the number of abstracts retrieved during the literature search. It is therefore not surprising that reviews which did not search Chinese language databases were more likely to have inconclusive results.

It was, however, not the purpose of our paper to comment on the effectiveness of acupuncture. Rather we wished to investigate the consistency of the literature search and data synthesis in production of Cochrane reviews relating to acupuncture. As the mission of the Cochrane Collaboration is to perform systematic reviews, it has set up criteria and guidelines for all Cochrane authors to follow. These guidelines include everything from how to define research questions to transparent search strategies to methods for analysing results. Lack of adherence to Cochrane Handbook guidelines for search strategies has been previously described72 and may explain some of the heterogeneity we observed. However, given the international focus of the Cochrane Collaboration especially, the proportion of English language only acupuncture reviews is surprising. Searching English language databases only may limit the number of trials retrieved for inclusion in systematic reviews.2

The limitations of this study must be acknowledged. Many reviews did not explicitly comment on language restrictions in their search strategy. Although we assumed that reviews that did not incorporate Chinese databases had employed a language restriction, it is possible that some non-English articles were retrieved in their search that were translated. We therefore might have overestimated the proportion of reviews with a language restriction. The results for which databases were searched, however, are unlikely to have been misclassified. This study did not capture the number of articles included in each of the reviews retrieved, only the conclusions. While the results are important, future studies should assess the association between language of literature databases searched and the number of included trials in acupuncture reviews.

In conclusion, specific criteria for Chinese and other non-English language search strategies need to be developed from within the Cochrane Collaboration to ensure the continued publication of high-quality reviews. The increasing interest in performing acupuncture-related reviews over the past 10 years makes the need for such guidelines even more pressing. In order to ensure that all of the available scientific literature is captured, systematic reviews focusing on acupuncture need to systematically search Chinese language databases as well as English databases in order to prevent the biases introduced by language and database restrictions.

Summary points

  • ▶. Systematic reviews should, by definition, include studies published in all languages.

  • ▶. Among Cochrane reviews of acupuncture, only 40% searched Chinese databases.

  • ▶. Exclusion of Chinese studies is associated with biased conclusions.

References

View Abstract

Footnotes

  • Competing interests None.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.