- Journal List
- J Gen Intern Med
- v.13(9); 1998 Sep
- PMC1497008
As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsem*nt of, or agreement with, the contents by NLM or the National Institutes of Health.
Learn more: PMC Disclaimer | PMC Copyright Notice
J Gen Intern Med. 1998 Sep; 13(9): 600–606.
PMCID: PMC1497008
PMID: 9754515
A Meta-Analysis Including Unpublished Data
Malcolm Man-Son-Hing, MD, MSc,1,2 George Wells, MSc, PhD,1,2 and Anita Lau2
Author information Copyright and License information PMC Disclaimer
Abstract
OBJECTIVE
With respect to the use of quinine for the treatment of nocturnal leg cramps, to determine whether the findings of a previously performed meta-analysis of published data are altered with the addition of unpublished data, and whether publication bias is present in this area.
DESIGN
A meta-analysis of eight (four published and four unpublished) randomized, double-blind, placebo-controlled trials, seven of which had a crossover design.
SETTING
Randomized trials that were available as of July 1997.
SUBJECTS
Ambulatory patients (659) who suffered from regular nocturnal leg cramps.
MAIN RESULTS
When individual patient data from all crossover studies were pooled, persons had 3.60 (95% confidence interval [CI] 2.15, 5.05) fewer cramps in a 4-week period when taking quinine compared with placebo. This compared with an estimate of 8.83 fewer cramps (95% CI 4.16, 13.49) from pooling published studies alone. The corresponding relative risk reductions were 21% (95% CI 12%, 30%) and 43% (95% CI 21%, 65%), respectively. Compared with placebo, the use of quinine was associated with an increased incidence of side effects, particularly tinnitus. Publication bias is present in the reporting of the efficacy of quinine for this indication, as almost all published studies reported larger estimates of its efficacy than did unpublished studies.
CONCLUSIONS
This study confirms that quinine is efficacious in the prevention of nocturnal leg cramps. However, its benefit may not be as large as reported from the pooling of published studies alone. Given the side effect profile of quinine, nonpharmacologic therapy (e.g., regular passive stretching of the affected muscle) is the best first-line treatment. For persons who find this ineffective and whose quality of life is significantly affected, a trial of quinine is warranted. Prescribing physicians must closely monitor the risks and benefits in individual patients. Publication bias is present in this area even though there is controversy about the role of quinine in the treatment of leg cramps. To minimize the possibility of this bias, persons performing medication-related meta-analyses should seek high-quality unpublished data from drug regulatory agencies and pharmaceutical companies.
Keywords: quinine, leg cramps, publication bias, meta-analysis
Nocturnal leg cramps are painful involuntary muscle contractions that usually occur while the patient is recumbent. They are a common condition, occurring in 70% of elderly persons at one time or another.1 Many nonpharmacologic and pharmacologic treatments have been proposed,2 the most studied therapy being quinine taken at bedtime. In 1995, we published a meta-analysis examining the short-term efficacy of quinine (up to 4 weeks) for the treatment of nocturnal leg cramps.3 The main study results combined data from four published randomized, double-blind, placebo-controlled crossover trials.4–7 Despite review of relevant articles and textbooks, and communication with authorities on the subject, we were unable to locate any unpublished studies that met our inclusion criteria. The results of the original meta-analysis showed that quinine was efficacious in the prevention of nocturnal leg cramps but did not affect the severity or duration of cramps. Insufficient data were available to draw conclusions about the side effect profile of quinine.
There is considerable controversy about the need to search for relevant unpublished trials for inclusion in meta-analyses.8 Some meta-analysts suggest that every attempt should be made to locate relevant unpublished studies because the added data will only augment the results.9 The contrary perspective is that it is not worthwhile to attempt to search for these studies as there is no systematic way of locating all relevant literature.10 Moreover, unpublished studies may be of inferior methodologic quality, and thus their results could be less reliable.
After publication of our original meta-analysis,3 it came to our attention that the United States Food and Drug Administration (FDA) possessed relevant unpublished information pertaining to the efficacy of quinine for nocturnal leg cramps.11 From 1974 to 1986, while preparing to issue a final regulation concerning the use of quinine for nocturnal leg cramps as a prescription product,12 the FDA accepted submissions from interested parties. We recently obtained copies of all relevant studies in their possession.13–17 The objectives for this analysis were to determine whether the inclusion of data from these unpublished studies can provide more precise estimates of the efficacy and side effect profile of quinine when used for the treatment of nocturnal leg cramps, or significantly alter the findings of the original meta-analysis, and whether publication bias is present in this area.
METHODS
Identification of Relevant Clinical Studies
Published Studies.
To identify any trials published since the completion of our original meta-analysis, a computerized literature search (MEDLINE and EMBASE) was repeated using the same key words (quinine, muscle cramps, and legs) for the time frame April 1994 to July 1997. Also, Current Contents was searched for the period January 1997 to July 1997. Authorities were again asked about any other published and unpublished studies that may be relevant.
Unpublished Studies.
The FDA report pertaining to quinine and leg cramps was reviewed,12 and the FDA was asked to forward all relevant dockets to us. The British and German drug regulatory bodies were contacted via telephone and mail requesting all relevant information in their possession. Pharmaceutical companies with previous and present involvement in the sale and distribution of quinine were also contacted, again asking if they possessed any relevant information.
All other trials found by the search procedure were assessed for fulfilment of the inclusion criteria. Our primary analysis involved pooling of studies that met the inclusion criteria of the original meta-analysis: randomization of patients, double-blind, placebo-controlled, crossover design, and general ambulatory patients. In an effort to combine data from parallel group studies and crossover trials, we performed a second meta-analysis that combined individual patient data from parallel group studies and those from the first treatment period of the crossover studies.
Data Extraction
In a process identical to the original meta-analysis, the following information was obtained from the eligible studies: number of patients, gender, age range, length of treatment period, presence of washout period, presence of side effects, and outcome measures used. Data were extracted by two independent assessors. Any disagreement was settled by collaborative review.
Meta-Analysis
Using a methodology identical to the original meta-analysis, for each eligible study, individual patient data were used to calculate a point estimate and 95% confidence interval (CI) for the efficacy of quinine compared with placebo for the following outcomes: reduction in the number of nocturnal leg cramps in a 4-week period (absolute and relative risk reduction), the severity of nocturnal leg cramps, and the duration of nocturnal leg cramps. Relative risk reduction (RRR) was calculated according to the following formula: [Number of cramps (Placebo) − Number of cramps (Quinine)]/Number of cramps (Placebo).
One-way analysis of variance was used to test for hom*ogeneity. This technique was also used to combine individual patient data from all relevant studies to produce an overall estimate and 95% CIs for the outcome measures above. Side effect data were also pooled from all eligible studies. McNemar's test was used to analyze paired side effect data. Regression analysis was performed to assess the relation of individual study estimates of the efficacy of quinine (as given by their RRR) to their length of treatment period and to sample size.
RESULTS
Systematic Overview
The computerized search identified two other possibly relevant published trials.18, 19 From the FDA, individual patient data from five unpublished clinical trials 13–17 relevant to the meta-analysis were obtained. From the inquiries to other drug regulatory agencies and pharmaceutical companies, no other relevant clinical trials were obtained.
Of the newly found trials, neither of the two published and only three of the five unpublished studies 13–15 met the inclusion criteria. From the original submissions to the FDA, individual patient data were available for the three eligible unpublished trials. Characteristics of the excluded trials are displayed in Table 1
Table 1
Newly Found Trials Not Meeting Inclusion Criteria
Open in a separate window
Adherence to the data extraction process resulted in complete agreement between the two assessors. The characteristics of all studies that met the inclusion criteria are summarized in Table 2.
Table 2
Relevant Characteristics of Trials Included in Meta-Analysis
Open in a separate window
Meta-Analysis
Crossover trials.
All eligible studies reported as an outcome measure the absolute change in the number of nocturnal leg cramps while the subjects were taking quinine compared with placebo. Because studies used treatment periods with varying lengths of time, we applied the same procedure as in the original meta-analysis to the studies that did not have treatment periods of 4 weeks in order to standardize the number of cramps experienced in a 4-week period. The standardization method used was to increase proportionately the number of cramps experienced by individual patients in studies with treatment periods of less than 4 weeks. For example, if a study had 2-week treatment periods, the number of cramps experienced in each period was doubled to simulate a 4-week treatment period. No study had treatment periods longer than 4 weeks.
All point estimates of the benefit of quinine from individual trials support its efficacy (Fig. 1). Published trials, compared with unpublished trials, consistently reported larger point estimates for the efficacy of quinine. Combining individual patient data from all trials showed 3.60 fewer cramps (95% CI 2.15, 5.05) in a 4-week period when taking quinine compared with placebo. The estimate of the benefit of quinine derived from data from published trials (8.83 fewer cramps; 95% CI 4.16, 13.49) was larger than that derived from data from unpublished trials (2.45 fewer cramps; 95% CI 1.03, 3.87). Thus, the addition of data from unpublished trials to the original meta-analysis of published data reduced the estimate of quinine's efficacy, but this result remained statistically significant (p= .0008).
Figure 1
Absolute reduction in nocturnal leg cramps (95% CI) in patients taking quinine compared with placebo (for standardized 4-week treatment period).
Inclusion of results from parallel group studies.
Of the studies meeting all inclusion criteria except crossover design, we were able to obtain individual patient data from one,17 but not the others.19, 20 Combining the individual patient data from this study with those from the first treatment periods of crossover studies gave an estimate of 2.87 fewer cramps (95% CI 0.20, 5.54) when taking quinine compared with placebo.
Relative Risk Reduction.
Using the baseline rate of nocturnal leg cramps when taking placebo, we calculated RRRs attributable to quinine (Table 3). The RRR derived from combining data from published studies (43%; 95% CI 21%, 65%) was larger than that derived from unpublished studies (21%; 95% CI 12%, 30%).
Table 3
Relative Risk Reductions for Quinine Compared with Placebo
Open in a separate window
Regression analysis showed that the longer the treatment period of a study, the larger its estimate of the RRR provided by quinine (p= .01). Also, a similar trend was found in the relation between RRR and the sample size of individual studies, but this did not reach statistical significance (p= .21).
Severity.
Six studies (three published and three unpublished) measured the severity of individual cramps. Methods of measuring cramp severity differed between studies, with two using 10-cm visual analogue scales and the rest using a 3-point scale (1, mild; 2, moderate; 3, severe). Using the same standardization technique as in the original meta-analysis, we converted the 10-cm visual analogue data to a 3-point scale by assigning values from 0 to 3.3 a score of 1, 3.4 to 6.6 a score of 2, and 6.7 to 10.0 a score of 3.
Combining the results of these trials (n= 378) showed that quinine reduced the severity of leg cramps (0.13 units; 95% CI 0.05, 0.21), with the results reaching statistical significance (p= .0023).
Duration.
Duration of cramps was not measured in any of the unpublished studies. Therefore, no further analysis for this measure was possible. The original meta-analysis showed quinine did not appear to reduce the duration of individual cramps, but this estimate was imprecise.
Side Effects.
Addition of individual patient data from the unpublished trials allowed a more comprehensive analysis of side effect data than was possible in the original meta-analysis.Table 4 shows the pooling of the side effect data, including those from an unpublished parallel group study supported by CIBA Pharmaceuticals.17 Subjects taking quinine reported more side effects than those taking placebo, and were more likely to withdraw from studies. The subjects taking quinine who had side effects reported a higher average number of side effects compared with those taking placebo.
Table 4
Side Effects Data*
Open in a separate window
To analyze the side effects data in the most statistically powerful manner, we examined their frequency in subjects who participated in crossover trials, thus providing paired data. However, for subjects who withdrew from studies because of side effects, we were not able to ascertain whether they withdrew during the first or second period. Therefore, we were able to analyze the frequency of side effects only in those subjects who did not withdraw.Table 5 categorizes individual side effects according to their presence when subjects were taking and not taking quinine. This analysis showed that tinnitus was the only individual side effect that occurred with a significantly higher frequency when subjects took quinine compared with placebo (p= .01). For the two subjects who withdrew from studies because of tinnitus, both were taking quinine at the time of withdrawal. Therefore, this result is significant regardless of our inability to determine whether these subjects also suffered tinnitus when taking placebo.
Table 5
Paired Side Effects Data for Patients Completing Crossover Trials (n= 409)
Open in a separate window
DISCUSSION
The role of quinine in the treatment of nocturnal leg cramps is controversial.21 Also, there is uncertainty whether data from unpublished studies should be included in meta-analyses. Therefore, we conducted this study to determine the impact of additional unpublished data on a meta-analysis that used published data to examine the efficacy of quinine in this condition, and to determine whether publication bias is present in this area.
Combining the results of four published and three unpublished crossover trials (n= 409), quinine compared with placebo reduced the absolute number of nocturnal leg cramps by 3.60 during a 4-week period (95% CI 2.15, 5.05), corresponding to a RRR of 21% (95% CI 12%, 30%). This compared with an estimate of 8.83 fewer cramps (95% CI 4.16, 13.45) and a 43% RRR (95% CI 21%, 65%) derived from pooling published studies only. Quinine, compared with placebo, was associated with a higher overall incidence of side effects, especially complaints of tinnitus.
For the seven crossover studies, the test for hom*ogeneity for the absolute reduction in cramps was significant (p= .001). As in the original meta-analysis, owing to its high estimate of the absolute benefit of quinine, the study of Connolly et al.5 was responsible for this heterogeneity. This study was the only one with exclusively male subjects and used the largest total dose of quinine. These may be reasons for its inconsistent results when compared with other studies. With removal of the Connolly et al. study from the analysis, the test for hom*ogeneity for the remaining six studies was not significant (p= .86), with a resulting estimate of the efficacy of quinine of 2.62 fewer cramps (95% CI 1.32, 3.91) in a 4-week period.
The results of this study show the limitation of the original meta-analysis, which pooled the results of a small number of trials, each of which enrolled a small number of patients. Its results may have been unduly influenced by a single study (Connolly et al.5) with results that are inconsistent with those of the other studies. The addition of unpublished data dampened the strong influence that the Connolly et al. study 5 had on the results of the original meta-analysis. Thus, the absolute benefit of quinine in this condition may not be as great as the original meta-analysis suggested.
Balanced against this interpretation is the potential confounding factor of our method of standardizing the length of treatment periods. As in the original meta-analysis, for studies with treatment periods less than 4weeks, we proportionately increased the number of cramps experienced by individual patients to simulate 4-week treatment periods. This method of standardization took a conservative approach to estimating the efficacy of quinine as regression analysis showed a direct relation between the length of treatment period in a study and its estimate of the efficacy of quinine. Therefore, as the treatment periods of all unpublished studies were of short duration (1 to 2 weeks), these trials may not have been of sufficient duration for quinine to show its full benefit. These results also highlight the need of a large randomized, controlled trial with longer treatment periods (at least 4 weeks).
The medical community is uncertain about the efficacy of quinine for nocturnal leg cramps. Therefore, regardless of their results, one might expect publication of all well-designed trials examining this issue. However, publication bias is problematic in this area as studies reporting larger estimates of the efficacy of quinine were more likely to be published. Similar problems may exist in other areas as well. How then should researchers performing medication-related meta-analyses proceed? The only feasible way of addressing this problem is the establishment of trial registries.22 However, until this occurs, pharmaceutical companies and drug regulatory agencies are potential sources of high-quality unpublished data. We believe that persons conducting these types of meta-analyses should routinely check with these sources.
In summary, the addition of high-quality data from unpublished trials to a meta-analysis of published data allowed us to estimate more precisely the efficacy of quinine for preventing nocturnal leg cramps. For treatment periods up to 4 weeks, this analysis confirmed that quinine is efficacious in the prevention of nocturnal leg cramps, though the magnitude of its effect may not be as great as previously reported.3 The results also showed that the short-term use of quinine is associated with an increased incidence of side effects, particularly tinnitus. Given this unclear benefit/risk ratio, nonpharmacologic management (e.g., regular passive stretching of affected muscles 23) should be attempted initially. If this fails and the patient's quality of life is significantly affected, then a trial of quinine (of at least 4 weeks) may be warranted. As with all potentially hazardous medications, physicians prescribing quinine need to closely monitor the benefits and risks in individual patients.
Acknowledgments
The authors thank J. Sidorov, MD, E. Shirley, MD, and P. Connolly, MD, for their original provision of individual patient data from their studies.
References
1. Hall AJ. Cramp and salt balance in ordinary life. Lancet. 1947;3:231–3. [PubMed] [Google Scholar]
2. Leclerc KM, Landry FJ. Benign nocturnal leg cramps. Current controversies over the use of quinine. Postgrad Med. 1996;99:177–84. [PubMed] [Google Scholar]
3. Man-Son-Hing M, Wells G. Meta-analysis of efficacy of quinine for treatment of nocturnal leg cramps in elderly people. BMJ. 1995;310:13–7. [PMC free article] [PubMed] [Google Scholar]
4. Sidorov J. Quinine sulfate for leg cramps: does it work? J Am Geriatr Soc. 1993;41:498–500. [PubMed] [Google Scholar]
5. Connolly PS, Shirley EA, Wassen JH, Nierenberg DW. Treatment of nocturnal leg cramps. A crossover trial of quinine vs. vitamin E. Arch Intern Med. 1992;152:1877–80. [PubMed] [Google Scholar]
6. Fung MC, Holbrook JH. Placebo-controlled trial of quinine therapy for nocturnal leg cramps. West J Med. 1989;151:42–4. [PMC free article] [PubMed] [Google Scholar]
7. Warburton A, Royston JP, O'Neill CJA, et al. A quinine a day keeps the leg cramps away? Br J Clin Pharmacol. 1987;23:459–65. [PMC free article] [PubMed] [Google Scholar]
8. Cook DJ, Guyatt GH, Ryan G, et al. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA. 1993;269:2749–53. [PubMed] [Google Scholar]
9. Easterbrook PJ, Berlin JA, Gopalan T, Matthews DR. Publication bias and dissemination of clinical research. Lancet. 1991;337:867–72. [PubMed] [Google Scholar]
10. Chalmers TC, Levin H, Sacks HS, Reitmen D, Berrier J, Nagalingam R. Meta-analysis of clinical trials as a scientific discipline, I: control of bias and comparison with large co-operative trials. Stat Med. 1987;6:315–25. [PubMed] [Google Scholar]
11. Nightingale SL. Quinine for nocturnal leg cramps. ACP J Club. 1995;123:86. Letter. [PubMed] [Google Scholar]
12. Drug products for the treatment and/or prevention of nocturnal leg muscle cramps for over-the-counter human use. Fed Reg. 1994;59:43234–52. [Google Scholar]
13. Leo Winter Associates, Inc . Final medical report and data summary analysis and final statistical report on double blind randomized crossover study of Q-VELR versus quinine sulfate versus vitamin E versus placebo in the treatment of nocturnal leg muscle cramps. Norwalk, Conn.: Bio-Products; 1986. Unpublished report. [Google Scholar]
14. Hays R, Goodman JJ. Clinical trial of the efficacy of quinine sulfate in the treatment of nocturnal leg muscle cramps, protocol 86-48. Memphis, Tenn.: Scholl Pharmaceuticals; 1986. Unpublished report. [Google Scholar]
15. Bottner M. Clinical trial of the efficacy of quinine sulfate in the treatment of nocturnal leg muscle cramps, protocol 84-46. Memphis, Tenn.: Scholl Pharmaceuticals; 1984. Unpublished report. [Google Scholar]
16. Clinical evaluation of Q-VEL in patients with nocturnal leg muscle cramps. New York, NY: Bio-Products; 1984. Unpublished report. [Google Scholar]
17. A short-term, randomized, double-blind, parallel study of Q-Vel vs. quinine sulfate vs. vitamin E vs. placebo in the prevention and treatment of nocturnal leg cramps. Edison, NJ: CIBA Pharmaceuticals; 1988. Unpublished report. [Google Scholar]
18. Maule B. Nocturnal cramps: quinine versus folklore. Practitioner. 1990;234:420–1. [PubMed] [Google Scholar]
19. Jansen PHP, Veenhuizen KCW, Verbeek ALM, Staatman H. Efficacy of hydroquinine in preventing frequent ordinary muscle cramp outlasts actual administration. J Neurol Sci. 1994;122:157–61. [PubMed] [Google Scholar]
20. Gorlich HD, von Gablenz E, Steinberg HW. Treatment of nocturnal leg cramps. A multi-center, double blind, placebo controlled comparison between the combination of quinine and theophylline ethylene diamine with quinine. Arzneimittelforschung. 1991;411:167–75. [PubMed] [Google Scholar]
21. Mackie MA, Davidson J. Prescribing of quinine and cramp inducing drugs in general practice. BMJ. 1995;311:1541. [PMC free article] [PubMed] [Google Scholar]
22. Felson DT. Bias in meta-analytic research. J Clin Epidemiol. 1992;45:885–92. [PubMed] [Google Scholar]
23. Daniell HW. Simple cure for nocturnal leg cramps. N Engl J. Med. 1979;301:216. [PubMed] [Google Scholar]
Articles from Journal of General Internal Medicine are provided here courtesy of Society of General Internal Medicine
