Systematic review of homeopathy: Questions and 
answers

27.09.2024, Harald J. Hamre* (1), Klaus von Ammon (2), Anja Glockmann (1), Helmut Kiene (1)

(1) Institute for Applied Epistemology and Medical Methodology at the Witten/Herdecke University,
Zechenweg 6, 79111 Freiburg, Germany *harald.hamre@ifaemm.de  www.ifaemm.de

(2) Rebbergstrasse 7, 8712 Stäfa, Switzerland

-->PDF

Here we answer 31 questions about our systematic review on the efficacy of homeopathy:

Hamre HJ, Glockmann A, von Ammon K, Riley DS, Kiene H. Efficacy of homoeopathic treatment: Systematic review of meta-analyses of randomised placebo-controlled homoeopathy trials for any indication. Syst Rev 2023; 12(191). https://doi.org/10.1186/s13643-023-02313-2 

 

Background

Q 1. Why was this systematic review conducted?

Starting point for the systematic review was the widespread assumption that effects of homeopathy are comparable to those of placebo. To test this hypothesis, all randomised, placebo-controlled homeopathy trials for any indication can be collected in a meta-analysis, assessing whether or not homeopathy has an effect beyond placebo. Since 1997, at least six such meta-analyses have been published. They differed in their methods for trial inclusion, assessment of risk of bias and data synthesis; furthermore, their results and conclusions differed to some extent. In order to analyse these differences and to create an overall synthesis of the results of these meta-analyses, we conducted a systematic review of them.

 

Q 2. Why did you not conduct a new meta-analysis of clinical trials instead of a systematic review? 

Before undertaking a new, updated meta-analysis, an overall assessment of the previous meta-analyses is helpful, especially since the two largest of the previous meta-analyses (Linde 1997Shang 2005) came to different conclusions. In addition, in the two most recent meta-analyses, trials on individualised homeopathy (Mathie 2014) and non-individualised homeopathy (Mathie 2017) were assessed separately. For a new meta-analysis, the question arises whether these homeopathy types should be examined together or separately. Thus a systematic review of the existing meta-analyses seemed warranted.

 

Methodology

Q 3. What is a systematic review?

A systematic review is an overview of all research that has been conducted on a particular topic, using a specific methodology. A classic example are systematic reviews of clinical trials on a particular intervention for a specific disease. For such systematic reviews, there are structured, criteria-based methods for search, identification, inclusion and analysis of the trials (more on this at ‎Q 6). The review leads to a synthesis of the most important results, typically with a vote counting: "X of Y trials had the result Z". Here, "Z" can be, for example, "a significant effect of homeopathy compared to placebo".

 

Q 4. What is the difference between systematic reviews and meta-analyses?

Many systematic reviews are also meta-analyses. In meta-analyses, the data synthesis contains not only the vote counting statement “X of Y trials showed a significant effect of Z”, but also a pooled estimate of the effect size for the “Y trials”. In this meta-analytic effect estimate, the number of patients as well as the effect size and its dispersion for each trial are taken into account. In this respect, meta-analyses are more informative than the quotient “X of Y trials”. However, a meta-analytic effect estimate is only possible when trial results are available in a suitable form.

 

Q 5. Your homeopathy review was a systematic review of meta-analyses. What does that 
mean?

The design was a 'systematic review of systematic reviews', also called 'umbrella review'. Our review was limited to systematic reviews with meta-analyses. Our research subject was therefore not the individual trials, but the meta-analyses thereof. We did not conduct a new meta-analysis; our data synthesis was the vote counting "X of Y meta-analyses had Z", where "Z" was not the results of the individual trials, but the pooled effect estimates of the individual meta-analyses.

As usual, we also summarised characteristics of the individual trials that had been described in the respective meta-analyses (see also ‎Q 24), but these data played only a minor role.

 

Q 6. Are there rules for preparing a systematic review?

Yes. There are currently four standards covering different aspects of systematic reviews that should be followed:

  • for the protocol of the systematic review: PRISMA-P,
  • for the publication manuscript: PRISMA-2020,
  • for assessing the methodological quality / risk of bias of the objects of investigation (here, the meta-analyses) several instruments are available; we used ROBIS,
  • for the assessment of confidence in cumulative evidence (also called ‘quality of evidence’): GRADE.

We have followed these standards.

 

Q 7. What is a protocol for a systematic review and why is it needed?

A protocol is a kind of blueprint for the planned systematic review. The PRISMA-P checklist contains 26 individual protocol items. The protocol must include, among other things:

  • the research question ‎(Q 8)
  • the eligibility criteria for the objects of investigation (here: the meta-analyses, ‎Q 9)
  • the primary outcome (‎Q 11)
  • the technical handling of data (search, inclusion, extraction, analysis, synthesis)

In the protocol the methods of the systematic review should be described in as much detail as possible. The protocol should be registered in advance in a public register for systematic reviews (see our protocol record and the full protocol). Later readers of the published systematic review can then compare the publication with the protocol and check to what extent the protocol specifications were implemented and whether relevant deviations from the protocol were adequately explained.

 

Q 8. What were the research questions?

The basic question was: Does homeopathy work better than placebo or not? To this end, we formulated two specific research questions (answers at ‎Q 21):

  • The first research question was: Does homeopathy have positive effects beyond placebo – in meta-analyses of randomised, placebo-controlled homeopathy trials for any indication?
  • The second research question was, in simple terms, to what extent does the first question make sense? Is it meaningful to speak of a common effect – or absence thereof – across different types of homeopathy (e.g., individualised, clinical, or complex homeopathy) and across different types of indications (e.g., acute, chronic)? The corresponding analysis finding is called ‘statistical homogeneity / heterogeneity of trial results’.

 

Q 9. Which meta-analyses were included in this systematic review? 

Meta-analyses of randomised, placebo-controlled homeopathy trials for any indication in humans were included. Meta-analyses without assessment of therapeutic benefit and meta-analyses restricted to specific indications were excluded. In addition, there were some technical criteria, including the minimum size of the research reports.

 

Q 10. Why were other meta-analyses and systematic reviews not included in this systematic review?

Because the meta-analyses with the above characteristics are best suited to assess the underlying question of the efficacy of homeopathy compared to placebo. The following would not be suitable:

  • meta-analyses of trials without placebo control groups (because in such trials no comparison with placebo is possible)
  • meta-analyses restricted to specific indications (because such meta-analyses do not provide valid information on the overall efficacy of homeopathy)
  • systematic reviews without meta-analysis (because such systematic reviews yield less informative results than reviews with pooled effect estimates, ‎Q 4).

 

Q 11. What were the most important data you took from the meta-analyses?

These data are called primary outcomes. According to the first research question (‎Q 8), these were the pooled effect estimates for homeopathy compared to placebo that had been calculated in the meta-analyses. For each meta-analysis, we included effect estimates for two sets of trials, where available:

  • all included trials in each meta-analysis: our first primary outcome analysis
  • the subset of trials with higher methodological quality / lower risk of bias (see ‎Q 21): our second primary outcome analysis

 

Q 12. What did you evaluate as authors?

Two of the four standards for systematic reviews (‎Q 6) concern evaluations by the authors, performed by us:

  • the risk of bias of the meta-analyses (we mainly used ROBIS for this purpose; results at ‎Q 20)
  • the confidence in cumulative evidence for a positive effect of homeopathy beyond placebo (according to the GRADE system; results at ‎Q 22)

 

Q 13. Why is the publication of your systematic review so long?

Our publication consists of the main document and five additional documents, totalling about 120 pages. All the data collected from the respective meta-analyses plus our assessments result in a lot of material. The PRISMA 2020 guidelines on systematic reviews recommend publishing all collected data, so that nothing is withheld from readers. Thus, knowledgeable readers can check what the analyses and conclusions are based on. We have followed this recommendation.

 

Results

Q 14. How many clinical trials were evaluated in the six meta-analyses? 

The six meta-analyses comprised 17, 18, 22, 54, 89 and 110 trials, respectively, with data available for pooled effect estimates. Adding these numbers together yields a total of 310 trials. Notably, some trials were included in more than one meta-analysis (depending on the eligibility criteria of the respective meta-analysis and the time of publication). Thus, if each trial used in several meta-analyses is counted only once, we counted 182 different trials or trial evaluations (cp. Additional file 4, Suppl. Table 15).

 

Q 15. From which period do the meta-analyses and the clinical trials included therein come?

The meta-analyses were published between 1997 and 2017; the clinical trials were from 1943 to 2014.

 

Q 16. Which countries did the trials come from?

According to the available data from three meta-analyses, the trials came from 18 different countries, the most frequently represented being United Kingdom, Germany, United States, India and France.

 

Q 17. What is the range of sample sizes in the clinical trials included?

The minimum number of participants per trial was between 5 and 28, the maximum number between 175 and 1573 participants.

 

Q 18. Which types of homeopathy were represented in the trials of the six meta-analyses?

  • Two meta-analyses (Linde 1998Mathie 2014) were restricted to trials on individualised or classical homeopathy. 
  • One meta-analysis (Mathie 2017) was restricted to non-individualised homeopathy, which was mostly divided into clinical homeopathy, complex homeopathy and isopathy. 
  • Three meta-analyses (Linde 1997Cucherat 2000Shang 2005) included trials on all homeopathic types (individualised and non-individualised), with the proportion of individualised homeopathy being 15-18%. Among the non-individualised homeopathy types, clinical homeopathy and complex homeopathy were the most frequent (Table 7).

 

Q 19. How was the methodological quality of the clinical trials assessed in the meta-analyses?

The methodological quality / risk of bias of randomised, placebo-controlled trials is measured by specific characteristics or components. A number of components have been used. The 10 most frequent components in the meta-analyses of our review are listed in Table 8. Among these, the most important components are the implementation of randomisation and the blinding of participants and researchers with regard to the administration of the test drug or placebo (No. 1-3 in Table 8).

For our systematic review, the classification of trials of higher methodological quality was particularly important (‎Q 21). We defined ‘high-quality trials’ according to three criteria, all of which had to be fulfilled:

  • The authors of the meta-analysis had explicitly named such a category (e.g. 'high quality', 'reliable evidence') and defined it.
  • Maximum one single high-quality category was defined for the respective meta-analysis.
  • The classification as 'high-quality' was based on at least 3 specified quality components.

According to these criteria, the category 'high quality trials' was documented in 4 of the 6 meta-analyses. The respective criteria were based on 8 components in Linde 1997 and 7 components in Mathie 2014 and Mathie 2017. In Shang 2005 it remained unclear whether 3 or 4 components had been used (Table 5).

The proportion of 'high-quality' trials was 6%, 14%, 19% and 29%, respectively (Table 8). In three meta-analyses, we were able to compare the quality of the homeopathy trials with the quality of other trials: the other trials were from all areas of medicine, with the same design, from a comparable time period and evaluated according to the same criteria. The proportion of 'high-quality' trials for homeopathy was significantly higher (homeopathy 19% vs. other trials 8% in Shang 2005, the most precise comparison) or similarly high (in Mathie 2014 and Mathie 2017, see Table 9) than in trials on other treatments. In terms of study quality, homeopathy is therefore not worse off than other medical interventions; there is generally room for improvement. By the way: for the question of efficacy, the results and the absolute number of 'high-quality' trials (‎Q 21) are more important than their proportion among all trials.

 

Q 20. What was your assessment of the risk of bias of the six meta-analyses?

The ROBIS tool contains assessments of 28 items and an overall assessment of risk of bias of the meta-analysis: 'low', 'unclear' or 'high'. According to these assessment criteria, three of the meta-analyses received the best rating 'low risk of bias' and three meta-analyses received the worst rating 'high risk of bias'. The three meta-analyses with low risk were the oldest meta-analysis (Linde 1997 with additional analyses in Linde 1999) and the two most recent meta-analyses (Mathie 2014 and Mathie 2017) (results of all assessments in Table 10, our comments on individual item assessments in Additional file 1).

Q 21. What were the main findings of your systematic review?

Regarding the first research question on the efficacy of homeopathy beyond placebo (see ‎Q 11):

  • First primary outcome analysis: 5 of the 6 meta-analyses included a pooled effect estimate for all included trials. All 5 showed a significantly positive effect of homeopathy compared to placebo.
  • Second primary outcome analysis: 4 meta-analyses included an effect estimate after sample restriction to high-quality trials (cf. ‎Q 19). In 3 of these 4 meta-analyses, the significant positive effect of homeopathy was retained, in 1 meta-analysis the positive effect was no longer significant (Table 12).

Regarding the second research question on a common homeopathy effect or not: 

  • A common positive effect (statistical homogeneity) was found for individualised homeopathy.
  • A common effect was not found for non-individualised homeopathy: In the meta-analyses with only or predominantly (82-85%) trials on non-individualised homeopathy, there was not one homogeneous effect, but several, different effects (statistical heterogeneity). Accordingly, effects of non-individualised homeopathy can vary in size and be significant or not significant, depending on, amongst others, the disease or the homeopathic remedy used (Additional file 3).

 

Q 22. How were your assessments of the confidence in cumulative evidence for efficacy of homeopathy compared to placebo?

For such assessments, the recommendations of the GRADE group (Grading of Recommendations Assessment, Development and Evaluation) are the leading standard. According to the specific questions of this systematic review, we used six publications by the GRADE group, each on a specific issue: Risk of bias of individual trialsInconsistency/heterogeneityRisk of publication bias/small study biasImprecisionIndirectness and Occasions for rating up the quality of evidence. According to the criteria stated there, the confidence in the cumulative evidence for homeopathy efficacy could be rated as 'high', 'moderate', 'low' or 'very low'.

The confidence in the cumulative evidence was rated as ‘high’ for individualised homeopathy (investigated in two meta-analyses), ‘moderate’ for non-individualised homeopathy (one meta-analysis) and ‘moderate’ for all homeopathy types (three meta-analyses). 

After restricting the sources of evidence to the three meta-analyses with low risk of bias (‎Q 20), the confidence in the cumulative evidence for all homeopathy types was now rated as 'high', the ratings for individualised and non-individualised homeopathy remained unchanged (detailed description in Additional file 3, more on this assessment in ‎Q 30).

Here, too, we compared with corresponding quality assessments from other systematic reviews: In an analysis of 608 Cochrane reviews on therapeutic procedures from all over medicine from 2013-2014, the confidence in the cumulative evidence for efficacy of the intervention, assessed for the primary outcome, was 'high' for only 13 percent, 'moderate' for 31 percent, 'low' for 32 percent and 'very low' for 24 percent of the interventions. Withstanding all limitations of such comparisons, the balance for homeopathy is remarkable.

Q 23. What were the main reasons for your ratings of confidence in the cumulative evidence?

Evidence from meta-analyses of randomised, placebo-controlled trials is generally considered to be the evidence with the lowest risk of bias.

For all three homeopathy types (individualised, non-individualised, all homeopathy types), the first primary analysis in all available meta-analyses showed a positive significant effect of homeopathy compared to placebo. Accordingly, the confidence in the cumulative evidence was initially rated as high.

  • For individualised homeopathy (two meta-analyses) there was no good reason to deviate from this rating.
  • For non-individualised homeopathy (one meta-analysis), the positive effect was no longer significant after sample restriction to high-quality trials; in addition, there were only three high-quality trials. Therefore, the confidence in the cumulative evidence was downgraded to 'moderate'.
  • For all homeopathy types (three meta-analyses), there were two problems with the Shang 2005 meta-analysis in this regard: First, no pooled effect estimate had been published for the 110 included homeopathy trials. Second, a subsequent bibliometric review (Mathie 2013) identified 41 additional trials that could have met the eligibility criteria for Shang 2005 but had not been identified by the authors. Thus, the evidence from this meta-analysis remained uncertain. Because Shang 2005 was the last and largest meta-analysis on all homeopathic types, the confidence in the cumulative evidence for all homeopathy types was downgraded from 'high' to 'moderate'.

 

Q 24. Could the inclusion of the same trials in several meta-analyses have distorted the results of your systematic review? 

No. The main result of our review (‎Q 21) was the proportion of meta-analyses with significantly positive effects of homeopathy over placebo, i.e. a quotient, not a sum or multiplication. Thus, using the same trials several times does not result in any additive or multiplicative magnification of effects, which would have meant a bias towards larger effects.

 

Q 25. But if the trials that were included in several meta-analyses had better results (more often statistically significant positive effects) by chance, couldn’t that lead to false-positive overall results?

That could be possible, but the trials included in several meta-analyses (‎Q 14) could also have worse results than the trials included only once. After all, the trials were not included in several meta-analyses because of their results, but because they met the respective eligibility criteria and had been published at the respective time. Accordingly, the proportion of trials with a significant positive homeopathy effect beyond placebo was

  • 36.5% (n = 113/310) when all trials in all meta-analyses were counted (cp. Publication: Interventions, results).
  • 36.3% (n = 66/182) when trials included in several meta-analyses were counted only once. 

The difference of 0.2% is irrelevant and statistically non-significant (p = 0.9444); there was therefore no bias.

 

Q 26. Can the positive results be explained by context and attention effects?

No. In all trials of the six meta-analyses included in our systematic review, the control patients received a placebo. Accordingly, context and attention effects, observation bias, etc. should have the same impact in intervention and control groups, while any additional effects can be attributed to the intervention, namely the homeopathic remedy.

 

Q 27. Are there any other limitations to your systematic review?

The following limitations apply:

  • The review was limited to meta-analyses on homeopathy for humans; homeopathy in veterinary medicine was not included.
  • For four assessed patient subgroups (adults, children, patients with acute or chronic disease) only sparse data were available.
  • In the meta-analyses, trials on homeopathic remedies for use in anthroposophic medicine, homotoxicology or radionics were excluded or only sparsely represented. The same applies to trials on prevention rather than treatment of existing symptoms.
  • The efficacy of homeopathy for specific indications was not object of our review.

 

Q 28. What were the most surprising findings for you when working on the review?

We were surprised by the comparisons regarding the methodological quality of the homeopathy trials (‎Q 19) and the confidence in cumulative evidence for efficacy of homeopathy (‎Q 22) with other trials and reviews, respectively. Quality deficiencies of homeopathy trials have often been commented on in the literature. The comparisons, however, showed the quality of homeopathy trials to be at least as good as for other trials; the same was the case for the confidence of the cumulative evidence for efficacy of homeopathy.

Another striking finding was the completely divergent risk of bias ratings of the six meta-analyses. Of three possible rating categories, three of the meta-analyses received the best rating, three the worst, and no meta-analysis received the middle rating. Particularly striking was the top ranking — not only of the two most recent meta-analyses from 2014 and 2017 but also of the oldest from 1997 (‎Q 20).

 

Conclusions

Q 29. What conclusions did you draw from your systematic review?

According to this systematic review, homeopathy can have positive effects beyond placebo in sick people. Notably, this systematic review is based on the best available evidence — meta-analyses of randomised, placebo-controlled trials — and was conducted and published in accordance with contemporary standards.

This new level of knowledge does not legitimize measures against homeopathy in health care.

Regarding further research, the implications of the systematic review differed with regard to the different homeopathy types:

  • For individualised, classical homeopathy, an update of the last meta-analysis of randomised placebo-controlled trials for any indication (Mathie 2014) would be useful.
  • For non-individualised homeopathy, the effects in our systematic review were statistically heterogeneous throughout. Accordingly, future meta-analyses on non-individualised homeopathy should focus on specific interventions and/or specific diseases.

 

Q 30. To what extent does this systematic review come to different assessments of the included clinical trials than the authors of previous meta-analyses? How can this be explained? 

Compared to the conclusions in the six meta-analyses, our conclusions are more clearly positive regarding the efficacy of homeopathy over placebo. There are several reasons for this:

  • The authors of each meta-analysis only evaluated their own results; we could also compare the methods and results of all six meta-analyses.
  • The authors of the respective meta-analyses only assessed the methodological quality of the included trials; we could also assess the methodological quality / risk of bias of the meta-analyses.
  • The most important reason: Although conclusions in scientific publications are supposed to be based on the most important results, the conclusions are usually subjective and context-dependent. This was different in our systematic review: Our conclusions on the efficacy of homeopathy were based on a criteria-based assessment of the cumulative evidence, according to GRADE. GRADE is a detailed, well-designed system with a good balance between rules, recommendations and individual assessments.

 

What happens next?

Q 31. Are there any relevant new clinical trials on homeopathy that are not included in previous meta-analyses?

Absolutely. The HOMIS study database at the Institute for Complementary and Integrative Medicine of the University of Berne contains information on randomised and other controlled clinical homeopathy studies. A new meta-analysis would be useful for individualised homeopathy (see ‎Q 29). For the period 2014-2021 (since the last meta-analysis on this topic by Mathie 2014), HOMIS lists an increase in 26 randomised, placebo-controlled trials on individualised homeopathy, 23 of which were published in peer-reviewed journals. How many of these would actually meet all eligibility criteria for a future systematic review and have results compatible with meta-analysis remains to be seen.

 

© Creative Commons CC BY-ND 4.0. 
https://creativecommons.org/licenses/by-nd/4.0/

Wir benötigen Ihre Zustimmung zum Laden der Übersetzungen

Wir nutzen einen Drittanbieter-Service, um den Inhalt der Website zu übersetzen, der möglicherweise Daten über Ihre Aktivitäten sammelt. Bitte überprüfen Sie die Details in der Datenschutzerklärung und akzeptieren Sie den Dienst, um die Übersetzungen zu sehen.