Article Text


Randomised clinical trials on acupuncture in the Korean literature: bibliometric analysis and methodological quality
  1. Sina Kim1,
  2. Hye Seon Sagong2,
  3. Jae Cheol Kong3,
  4. Jun-Yong Choi4,
  5. Myeong Soo Lee5,
  6. L Susan Wieland6,
  7. Eric Manheimer6,
  8. Byung-Cheul Shin2
  1. 1Department of Korean Medical Science, School of Korean Medicine, Pusan National University, Yangsan, South Korea
  2. 2Department of Korean Medicine, School of Korean Medicine, Pusan National University, Yangsan, South Korea
  3. 3Department of Rehabilitation Medicine, Chumdan Korean Medicine Hospital, Kwangju, South Korea
  4. 4Department of Internal Medicine, Korean Medicine Hospital, Pusan National University, Yangsan, South Korea
  5. 5Medical Research Division, Korea Institute of Oriental Medicine, Daejeon, South Korea
  6. 6Center for Integrative Medicine, University of Maryland School of Medicine, Baltimore, Maryland, USA
  1. Correspondence to Dr Byung-Cheul Shin, Division of Clinical Medicine, Department of Korean Medicine, School of Korean Medicine, Pusan National University, Yangsan 626-870, South Korea; drshinbc{at}; drshinbc{at}


Objective Acupuncture systematic reviewers have increasingly searched Chinese databases and journals to identify eligible randomised clinical trials (RCTs). However, reviewers have infrequently searched for eligible RCTs in Korean databases and journals. This study aimed to identify difficult to locate acupuncture RCTs in Korean databases and journals and to assess the characteristics and quality of the identified RCTs.

Methods Eleven electronic databases and seven journals were searched up to December 2012. All RCTs using needle acupuncture were considered for inclusion. Key study characteristics were extracted and risk of bias was assessed using the Cochrane Collaboration tool.

Results One hundred and forty-three publications met our inclusion criteria. Acupuncture RCTs in the Korean literature emerged in the mid-1990s and increased in the mid-2000s. Diverse methods of acupuncture were used, including some methods unique to Korea (eg, Saam acupuncture). The largest proportion of trials evaluated acupuncture for musculoskeletal conditions (27.3%). The mean sample size was 44.3±25.3 per trial. Random sequence generation methods were reported in 44.8% of the RCTs, whereas only 11.9% reported methods of allocation concealment. A low proportion of trials reported participant blinding (32.9%) and outcome assessment blinding (18.9%).

Conclusions Korean acupuncture trials, many of which evaluate acupuncture styles unique to Korea, are typically omitted from systematic reviews of acupuncture, resulting in the potential for language bias. The development of this database of difficult to locate Korean trials, which includes English language translations of abstracts, will enable these trials of varying quality to be assessed for inclusion in future acupuncture systematic reviews.

  • Acupuncture
  • Systematic Reviews

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See:

Statistics from


Acupuncture has an ancient history as a dominant therapeutic intervention in East Asian countries such as China, Korea and Japan.1 The therapy has been widely used for the treatment of various conditions. Acupuncture has recently been increasingly introduced and popularised in Western societies, thus increasing the demand for research on its efficacy. Currently, many clinical trials and reviews on acupuncture are listed in English-based medical databases such as MEDLINE and The Cochrane Library.

Systematic reviews are regarded as the strongest evidence among clinical studies in the field of evidence-based medicine because of the rigorous analytical methods used. The reviews use explicit systematic methods that are selected with a view to minimising bias.2 The identification of all the potentially eligible trials has been described as ‘the most fundamental challenge’ of preparing systematic reviews.3 It is therefore important to conduct an extensive literature search that covers all relevant articles to avoid publication, language and other possible biases.4

As acupuncture is still more prevalent in East Asian countries, a number of clinical trials on acupuncture have been conducted in the region and indexed only in local databases. Because these trials are often not included in MEDLINE, they may not be identified through literature searches and are therefore likely to be omitted from systematic reviews. In a previous study by Kim et al,5 75% of acupuncture-related Cochrane reviews included Chinese databases in their search, whereas only 4% of the reviews included Korean and Japanese databases. This lack of relevant database searches due to language barriers increases the risk of language bias.

To address this concern, there have been attempts to introduce information about Korean clinical studies on acupuncture into Western journals. A 2005 overview summarised ‘clinical case studies’ of acupuncture published in the Korean literature from 1983 to 2001.6 However, this overview included all type of clinical studies, did not report the numbers or characteristics of any randomised clinical trials (RCTs) identified and did not include assessments of the methodologies used in the identified studies. A more comprehensive review of RCTs on acupuncture published in Korea through May 2007 was conducted by our authors in 2009,7 with a more rigorous analytical method and systematic approach. However, the excessively rigid inclusion criteria for the identified articles prevented this review from covering many Korean RCTs. Indeed, because only 10 articles were included in that review, certain trends or tendencies in the publications could not be analysed.

For this reason, we aimed to update our previous work and to provide comprehensive information on Korean acupuncture RCTs to those who need this information. This paper is an extended sample of the Korean acupuncture RCTs from our previous work.7 The study aims to analyse the bibliometric characteristics and the quality of RCTs on acupuncture in Korean literature, with an effort to retrieve and include all of the relevant articles. English translations of the abstracts and PDFs of the full Korean articles of identified trials are contributed to the Cochrane Complementary Medicine Field specialised register of controlled trials, and trial citations are forwarded to The Cochrane Central Register of Controlled Trials (CENTRAL) to increase their accessibility to systematic reviewers. We systematically approached the literature search and selection process as well as the data extraction and quality assessment. Analysis of the characteristics of Korean acupuncture RCTs from this study will be of help to future researchers and will provide preliminary data for acupuncture researchers who are unable to access Korean literature directly.



Electronic bibliographic databases and journals

Eleven bibliographic databases were searched electronically and seven Korean journals were searched manually, all from their inception up to December 2012. Korean trials indexed in English-based databases were not considered. There could be studies which were indexed in English-based databases such as MEDLINE or EMBASE, but those sites were not searched. Key features of each database and journal searched are included in online supplementary appendix S1. The search terms used for the electronic databases were ‘(acupuncture OR acupuncture point OR meridian) AND (random OR control)’. Korean language terms relevant to acupuncture or clinical trials were also used.

Other sources

In addition to searching electronic sources, we searched our own paper files, data on clinical trial performances and information from clinical studies supported by the Korea Institute of Oriental Medicine (KIOM). To retrieve grey literature, dissertations and conference proceedings were searched. Finally, we checked the reference lists of relevant articles.

Study selection

Two authors (SK, JCK) conducted the literature search and retrieved citations of studies presumed to be potential RCTs. Full-text articles were obtained for all potentially eligible trials and two authors (SK, BCS) independently checked the eligibility criteria of the full-text articles.

Parallel or crossover RCTs that assessed the efficacy or physiological features of acupuncture were included, regardless of the participants’ medical conditions, language (Korean or English) of publication or publication year. Uncontrolled clinical trials and controlled but non-RCTs, including trials using a quasi-random method of allocation (eg, alternation, date of birth or case record number) were excluded. We predefined acupuncture as penetrating needling on acupuncture points, regardless of acupuncture methods or stimulation types; non-penetrative interventions such as acupressure, transcutaneous electrical nerve stimulator (TENS) or laser acupuncture were therefore excluded but reported separately for future research (see online supplementary appendix S2). We also excluded pharmacopuncture because the efficacy of needling was combined with injected pharmacologically active substances.8 Controls included sham/placebo acupuncture, other types of acupuncture, no treatment or other active intervention. We included RCTs in which co-interventions were combined with acupuncture in the experimental group if the co-interventions were also given equally to the control group(s). However, if the main purpose of the trial was to assess the efficacy of the co-intervention and not the acupuncture, the trial was excluded (see online supplementary appendix S3). All types of participants, even healthy subjects, were included because our research aim was to present all of the features of Korean acupuncture RCTs.

Data extraction and analysis

The characteristics of the included RCTs were analysed and extracted in the following categories: (1) publication year and journal for bibliometric analysis; (2) study design, acupuncture method (acupuncture point selection method), acupuncture stimulation type, structure of acupuncture experimental and control groups; (3) medical conditions of participants; (4) sample size; and (5) whether data on safety were reported. The medical conditions of participants were categorised according to a modification of the International Classification of Diseases 10.9

Risk of bias

Risk of bias was assessed independently by two authors (SK, JCK) using the Cochrane Collaboration's risk of bias tool which includes the following seven domains2: (1) random sequence generation; (2) allocation concealment; (3) blinding of participants and personnel; (4) blinding of outcome assessment; (5) incomplete outcome data; (6) selective reporting (if all prespecified outcomes in a prepublished protocol were reported, the risk of bias was judged low); and (7) other sources of bias. Discrepancies were resolved by consensus through discussion between the two reviewers based on the judgement criteria in Chapter 8 of the ‘Cochrane Handbook for Systematic Reviews of Interventions’2 and, if needed, by asking for further evaluation by a third reviewer (B-CS).


Study description

As shown in online supplementary appendix S4, our searches identified 994 citations of which 283 full-text articles were evaluated in full; 140 full-text articles were excluded, leaving 143 trials that met our inclusion criteria (131 published in Korean and 12 in English, see online supplementary appendix S5). The literature search process is summarised in online supplementary appendix S4, following the PRISMA flow diagram.10

Bibliometric analysis

The first acupuncture RCT published in Korea was in 1989,11 and a second RCT was published in 1996. From then on, the number of publications increased, with the highest number of publications occurring in 2007 (23 RCTs). Since then, there has been a fluctuation in the number of publications per year, with 14 published in 2012 (figure 1).

Figure 1

Number of Korean acupuncture randomised clinical trials by publication year.

The 143 RCTs were published across 17 journals; 128 of the 143 records were published in journals and 15 studies were identified from dissertations. The journals publishing acupuncture RCTs were mostly related to Traditional Korean Medicine (TKM) (11 of 17 journals, 121 of 143 papers; table 1).

Table 1

Number of Korean acupuncture randomised clinical trials published by journal

Study design, types of interventions and controls

The study designs, acupuncture styles, acupuncture stimulation methods and types of controls are shown in table 2. Parallel designs (n=131; 91.7%) were more commonly used than crossover design RCTs (n=12; 8.3%), with the two-arm parallel design predominating (n=100; 69.9%).

Table 2

Study design and types of interventions in Korean acupuncture randomised clinical trials

Diverse acupuncture styles were used in the RCTs, reflecting the acupuncture trends in Korea. Although classical acupuncture was used in most of the studies (n=104; 72.7%), methods unique to Korea such as Saam acupuncture and constitution acupuncture12 were used in 21 trials (14.7%).

Manual stimulation of the needle was the most frequently used form of stimulation for acupuncture (n=110; 76.9%), followed by electroacupuncture (n=25; 17.5%), intradermal acupuncture (n=4; 2.8%) and other techniques. Intradermal acupuncture refers to a specially made short needle that is embedded in the skin or subcutaneously for a relatively long time (less than 1–3 days, twice a week is general in clinical practice).

Controls included other types of acupuncture (to compare the effects of different acupuncture methods) (n=58; 3.7%), sham acupuncture (n=39; 24.7%), no treatment (n=32; 20.3%) or other active interventions (n=19; 12.0%). There were 10 RCTs (6.3%) in which another intervention was applied to both groups while acupuncture was applied only to the experimental group (table 2).

Medical conditions

Of the included trials, 74.1% were classified into one of 12 medical condition categories with the greatest concentrations being in the categories musculoskeletal conditions, circulatory conditions and nervous system conditions. The remaining 25.9% of trials were conducted in healthy participants (table 3).

Table 3

Categories of medical conditions in Korean acupuncture randomised clinical trials

Sample size

The total number of participants from the 143 included RCTs was 6332. Sample size per trial ranged from 8 to 175, with a mean ± SD of 44.3±25.3. Sample size per arm (total 330 arms) ranged from 4 to 116, with a mean ± SD of 19.3±11.9. Seven RCTs only stated the total sample size and did not specify the sample size of each group at the randomisation period. Thus, when calculating the mean and SD of the sample size per arm, we divided the total sample size by the number of arms for those seven trials.13–19 Statistical power analysis was conducted in only three studies (2.1%).20–22

Risk of bias

The risk of bias of the included RCTs is shown in table 4. A random sequence was adequately generated in 44.8% (n=64) of the included RCTs. Many studies did not specify how the random sequence was generated and they were estimated as unclear risk of bias; 88.1% of the RCTs (n=126) did not describe allocation concealment. Examples of participant blinding were sham needle or superficial needling on non-acupuncture points. The proportions of trials in which participants were blinded (n=47; 32.9%), outcome assessors were blinded (n=27; 18.9%) and incomplete outcome data were addressed (n=42; 29.4%) were low. Almost none of the trials were free of selective outcome reporting; only three protocols were published, which were estimated to have low risk of bias. Almost none of the trials were free of other risk of bias either; only two trials clearly stated on every method that was applied to minimise biases whenever applicable.

Table 4

Methodological quality of Korean acupuncture randomised clinical trials according to Cochrane risk of bias

Safety report

Only 23.8% (n=34) of the RCTs included descriptions of adverse events associated with acupuncture. Those that did report adverse events commented that events were mild or that there were no adverse events at all.


This study is an analysis of acupuncture RCTs published in the Korean literature. The trials assessed the effectiveness of various types of acupuncture in a variety of medical conditions. RCTs of acupuncture in the Korean literature slowly emerged in the mid-1990s and increased markedly in the mid-2000s. From the literature search, we found that there were substantial numbers of clinical trials on acupuncture in Korean literature, even though Korean trials indexed in English-based databases were not considered.

Although acupuncture RCTs in Korea have strength in quantity, the high risk of bias remains their weakness. As mentioned above, a random sequence was adequately generated in 44.8% of the included RCTs whereas only 11.9% concealed allocation. The proportions reporting double blinding or blinding of outcome assessments were low, as were the proportions addressing incomplete outcome data. Almost none of the included RCTs were free of selective outcome reporting. Studies with a high risk of bias are likely to distort the outcomes and exaggerate the treatment effects.23

RCTs published in other East Asian countries such as China and Japan have also been shown to have high risks of bias. In a study to investigate the adequacy of randomisation in RCTs published in Chinese journals,24 only 21.6% of Traditional Chinese Medicine (TCM) RCTs conducted at medical universities and 12.4% of RCTs conducted at level 3 hospitals were considered authentic RCTs. Similar results were shown in an earlier study25 which reported only 15% of the RCTs to be blinded. Even lower methodological quality was reported in a study using the Consolidated Standards of Reporting Trials (CONSORT) checklist26 which identified only 7.9% of TCM RCTs published in China to have reported methods of generating a randomisation sequence. Studies evaluating acupuncture clinical trials in Japan show similar results to those in Korea.27 Among 33 trials described as randomised, seven of the trials have been identified as quasi-randomised.

Although it remains controversial whether non-English trials, which may have a higher risk of bias, should be included in systematic reviews, previous studies suggest that there is no significant difference between English trials and non-English trials in terms of methodological quality.28 Thus, it should be recommended that systematic reviews include studies regardless of publication language because a core component of systematic reviews is to conduct an extensive literature search that covers all relevant articles. To dispel the controversy, well-designed RCTs on acupuncture with rigorous methodology must be conducted by Korean or other East Asian authors.

From this study, characteristics of Korean acupuncture were identified. Korea has developed individualised acupuncture treatment methods such as Saam, Taegeuk, Sasang constitution or 8-constitution acupuncture. Various needle stimulation methods, such as burning acupuncture or warm needling acupuncture, also contribute to the diversity of Korean acupuncture. This diversity was reflected in the study designs, where the most frequently used control design in the included RCTs compared one type of acupuncture with another type of acupuncture.

A great number of Korean databases and relevant journals were searched for this study, allowing the largest number of RCTs on acupuncture in Korean literature to be evaluated. Categorised information from this study may be of help to future researchers. The number of articles related to intervention type or various medical conditions can be used for research. The search can also be used to indicate directions for further research—for example, if trials on commonly used acupuncture interventions or in medical conditions commonly treated with acupuncture are insufficient, this may imply that research resources should be assigned to these areas.

There are several limitations to this study. First, in spite of our effort to retrieve all relevant articles, we cannot be certain that our search was all-inclusive. No standard search filter exists in Korean databases, hindering an efficient and complete search for the records. Therefore, a researcher should review papers one by one; this study is a summary of Korean acupuncture that was completed via a manual search.

Another limitation is that we could not draw conclusions on the efficacy of acupuncture from this study. As mentioned above, the most frequently used control design in the included RCTs was comparing one type of acupuncture with another type of acupuncture. Thus, evaluating the clinical effectiveness of acupuncture in these cases was impossible or not applicable. Another control design in the trials compared acupuncture with another intervention. In many of these trials the efficacy of the intervention used as a control or the other type of acupuncture used as a control has not been evaluated by RCTs, which prevents us from demonstrating the efficacy of the targeted acupuncture as well. To overcome this limitation, another control group such as a sham acupuncture group should be added. Lastly, because more than two interventions were simultaneously performed in some trials, it was not possible for these trials to demonstrate the effectiveness of acupuncture independent of other treatments.


This review has identified and evaluated the largest number of difficult to locate acupuncture RCTs in the Korean literature to date. Acupuncture systematic reviewers who cannot access Korean databases and journals might review the compiled list of these Korean RCTs (see online supplementary appendix S5) to identify any that are potentially eligible for their reviews. Citations of potentially eligible trials can also be accessed through searching the CENTRAL database in The Cochrane Library. These RCTs on acupuncture in the Korean literature have assessed the effectiveness of various types of acupuncture on a variety of medical conditions, reflecting the diverse application of acupuncture in Korean clinical practice. Although these RCTs have increased in number, there is great room for improvement in their methodological quality.

Summary points

  • Systematic reviews of acupuncture rarely include Korean literature.

  • We conducted an exhaustive search of databases.

  • We located 143 RCTs with 6332 participants; 73% used classical acupuncture.


We would like to thank Tae Hee Kang and Dahee Kim for their assistance with the literature search.


View Abstract

Review history and Supplementary material


  • Correction notice This article has been corrected since it was published Online First. Funding and Disclaimer statements have been included.

  • Contributors B-CS designed the study. SK and JCK conducted the literature search. SK and B-CS independently checked the eligibility criteria of the searched data. SK and HSS extracted and analysed the data. SK, JCK and B-CS assessed the risk of bias of the included data. SK drafted the manuscript. J-YC, MSL, LSW and EM revised the manuscript critically. B-CS the guarantor, approved the final manuscript. Tae Hee Kang and Dahee Kim assisted with the literature search.

  • Funding EM and LSW were partially funded by grant no. R24 AT001293 from the National Center for Complementary and Alternative Medicine (NCCAM) of the US National Institutes of Health.

  • Disclaimer The contents of this article are solely the responsibility of the authors and do not necessarily represent the official views of the NCCAM or the National Institutes of Health.

  • Competing interests None.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement The appendices are provided for future acupuncture studies. Supplementary data are available at Acupuncture in Medicine Online.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.