27.09.2024, Harald J. Hamre* (1), Klaus von Ammon (2), Anja Glockmann (1), Helmut Kiene (1)
(1) Institute for Applied Epistemology and Medical Methodology at the Witten/Herdecke University,
Zechenweg 6, 79111 Freiburg, Germany *harald.hamre@ifaemm.de www.ifaemm.de
(2) Rebbergstrasse 7, 8712 Stäfa, Switzerland
Here we answer 31 questions about our systematic review on the efficacy of homeopathy:
Hamre HJ, Glockmann A, von Ammon K, Riley DS, Kiene H. Efficacy of homoeopathic treatment: Systematic review of meta-analyses of randomised placebo-controlled homoeopathy trials for any indication. Syst Rev 2023; 12(191). https://doi.org/10.1186/s13643-023-02313-2
Background
Q 1. Why was this systematic review conducted?
Starting point for the systematic review was the widespread assumption that effects of homeopathy are comparable to those of placebo. To test this hypothesis, all randomised, placebo-controlled homeopathy trials for any indication can be collected in a meta-analysis, assessing whether or not homeopathy has an effect beyond placebo. Since 1997, at least six such meta-analyses have been published. They differed in their methods for trial inclusion, assessment of risk of bias and data synthesis; furthermore, their results and conclusions differed to some extent. In order to analyse these differences and to create an overall synthesis of the results of these meta-analyses, we conducted a systematic review of them.
Q 2. Why did you not conduct a new meta-analysis of clinical trials instead of a systematic review?
Before undertaking a new, updated meta-analysis, an overall assessment of the previous meta-analyses is helpful, especially since the two largest of the previous meta-analyses (Linde 1997, Shang 2005) came to different conclusions. In addition, in the two most recent meta-analyses, trials on individualised homeopathy (Mathie 2014) and non-individualised homeopathy (Mathie 2017) were assessed separately. For a new meta-analysis, the question arises whether these homeopathy types should be examined together or separately. Thus a systematic review of the existing meta-analyses seemed warranted.
Methodology
Q 3. What is a systematic review?
A systematic review is an overview of all research that has been conducted on a particular topic, using a specific methodology. A classic example are systematic reviews of clinical trials on a particular intervention for a specific disease. For such systematic reviews, there are structured, criteria-based methods for search, identification, inclusion and analysis of the trials (more on this at Q 6). The review leads to a synthesis of the most important results, typically with a vote counting: "X of Y trials had the result Z". Here, "Z" can be, for example, "a significant effect of homeopathy compared to placebo".
Q 4. What is the difference between systematic reviews and meta-analyses?
Many systematic reviews are also meta-analyses. In meta-analyses, the data synthesis contains not only the vote counting statement “X of Y trials showed a significant effect of Z”, but also a pooled estimate of the effect size for the “Y trials”. In this meta-analytic effect estimate, the number of patients as well as the effect size and its dispersion for each trial are taken into account. In this respect, meta-analyses are more informative than the quotient “X of Y trials”. However, a meta-analytic effect estimate is only possible when trial results are available in a suitable form.
Q 5. Your homeopathy review was a systematic review of meta-analyses. What does that
mean?
The design was a 'systematic review of systematic reviews', also called 'umbrella review'. Our review was limited to systematic reviews with meta-analyses. Our research subject was therefore not the individual trials, but the meta-analyses thereof. We did not conduct a new meta-analysis; our data synthesis was the vote counting "X of Y meta-analyses had Z", where "Z" was not the results of the individual trials, but the pooled effect estimates of the individual meta-analyses.
As usual, we also summarised characteristics of the individual trials that had been described in the respective meta-analyses (see also Q 24), but these data played only a minor role.
Q 6. Are there rules for preparing a systematic review?
Yes. There are currently four standards covering different aspects of systematic reviews that should be followed:
We have followed these standards.
Q 7. What is a protocol for a systematic review and why is it needed?
A protocol is a kind of blueprint for the planned systematic review. The PRISMA-P checklist contains 26 individual protocol items. The protocol must include, among other things:
In the protocol the methods of the systematic review should be described in as much detail as possible. The protocol should be registered in advance in a public register for systematic reviews (see our protocol record and the full protocol). Later readers of the published systematic review can then compare the publication with the protocol and check to what extent the protocol specifications were implemented and whether relevant deviations from the protocol were adequately explained.
Q 8. What were the research questions?
The basic question was: Does homeopathy work better than placebo or not? To this end, we formulated two specific research questions (answers at Q 21):
Q 9. Which meta-analyses were included in this systematic review?
Meta-analyses of randomised, placebo-controlled homeopathy trials for any indication in humans were included. Meta-analyses without assessment of therapeutic benefit and meta-analyses restricted to specific indications were excluded. In addition, there were some technical criteria, including the minimum size of the research reports.
Q 10. Why were other meta-analyses and systematic reviews not included in this systematic review?
Because the meta-analyses with the above characteristics are best suited to assess the underlying question of the efficacy of homeopathy compared to placebo. The following would not be suitable:
Q 11. What were the most important data you took from the meta-analyses?
These data are called primary outcomes. According to the first research question (Q 8), these were the pooled effect estimates for homeopathy compared to placebo that had been calculated in the meta-analyses. For each meta-analysis, we included effect estimates for two sets of trials, where available:
Q 12. What did you evaluate as authors?
Two of the four standards for systematic reviews (Q 6) concern evaluations by the authors, performed by us:
Q 13. Why is the publication of your systematic review so long?
Our publication consists of the main document and five additional documents, totalling about 120 pages. All the data collected from the respective meta-analyses plus our assessments result in a lot of material. The PRISMA 2020 guidelines on systematic reviews recommend publishing all collected data, so that nothing is withheld from readers. Thus, knowledgeable readers can check what the analyses and conclusions are based on. We have followed this recommendation.
Results
Q 14. How many clinical trials were evaluated in the six meta-analyses?
The six meta-analyses comprised 17, 18, 22, 54, 89 and 110 trials, respectively, with data available for pooled effect estimates. Adding these numbers together yields a total of 310 trials. Notably, some trials were included in more than one meta-analysis (depending on the eligibility criteria of the respective meta-analysis and the time of publication). Thus, if each trial used in several meta-analyses is counted only once, we counted 182 different trials or trial evaluations (cp. Additional file 4, Suppl. Table 15).
Q 15. From which period do the meta-analyses and the clinical trials included therein come?
The meta-analyses were published between 1997 and 2017; the clinical trials were from 1943 to 2014.
Q 16. Which countries did the trials come from?
According to the available data from three meta-analyses, the trials came from 18 different countries, the most frequently represented being United Kingdom, Germany, United States, India and France.
Q 17. What is the range of sample sizes in the clinical trials included?
The minimum number of participants per trial was between 5 and 28, the maximum number between 175 and 1573 participants.
Q 18. Which types of homeopathy were represented in the trials of the six meta-analyses?
Q 19. How was the methodological quality of the clinical trials assessed in the meta-analyses?
The methodological quality / risk of bias of randomised, placebo-controlled trials is measured by specific characteristics or components. A number of components have been used. The 10 most frequent components in the meta-analyses of our review are listed in Table 8. Among these, the most important components are the implementation of randomisation and the blinding of participants and researchers with regard to the administration of the test drug or placebo (No. 1-3 in Table 8).
For our systematic review, the classification of trials of higher methodological quality was particularly important (Q 21). We defined ‘high-quality trials’ according to three criteria, all of which had to be fulfilled:
According to these criteria, the category 'high quality trials' was documented in 4 of the 6 meta-analyses. The respective criteria were based on 8 components in Linde 1997 and 7 components in Mathie 2014 and Mathie 2017. In Shang 2005 it remained unclear whether 3 or 4 components had been used (Table 5).
The proportion of 'high-quality' trials was 6%, 14%, 19% and 29%, respectively (Table 8). In three meta-analyses, we were able to compare the quality of the homeopathy trials with the quality of other trials: the other trials were from all areas of medicine, with the same design, from a comparable time period and evaluated according to the same criteria. The proportion of 'high-quality' trials for homeopathy was significantly higher (homeopathy 19% vs. other trials 8% in Shang 2005, the most precise comparison) or similarly high (in Mathie 2014 and Mathie 2017, see Table 9) than in trials on other treatments. In terms of study quality, homeopathy is therefore not worse off than other medical interventions; there is generally room for improvement. By the way: for the question of efficacy, the results and the absolute number of 'high-quality' trials (Q 21) are more important than their proportion among all trials.
Q 20. What was your assessment of the risk of bias of the six meta-analyses?
The ROBIS tool contains assessments of 28 items and an overall assessment of risk of bias of the meta-analysis: 'low', 'unclear' or 'high'. According to these assessment criteria, three of the meta-analyses received the best rating 'low risk of bias' and three meta-analyses received the worst rating 'high risk of bias'. The three meta-analyses with low risk were the oldest meta-analysis (Linde 1997 with additional analyses in Linde 1999) and the two most recent meta-analyses (Mathie 2014 and Mathie 2017) (results of all assessments in Table 10, our comments on individual item assessments in Additional file 1).
Q 21. What were the main findings of your systematic review?
Regarding the first research question on the efficacy of homeopathy beyond placebo (see Q 11):
Regarding the second research question on a common homeopathy effect or not:
Q 22. How were your assessments of the confidence in cumulative evidence for efficacy of homeopathy compared to placebo?
For such assessments, the recommendations of the GRADE group (Grading of Recommendations Assessment, Development and Evaluation) are the leading standard. According to the specific questions of this systematic review, we used six publications by the GRADE group, each on a specific issue: Risk of bias of individual trials, Inconsistency/heterogeneity, Risk of publication bias/small study bias, Imprecision, Indirectness and Occasions for rating up the quality of evidence. According to the criteria stated there, the confidence in the cumulative evidence for homeopathy efficacy could be rated as 'high', 'moderate', 'low' or 'very low'.
The confidence in the cumulative evidence was rated as ‘high’ for individualised homeopathy (investigated in two meta-analyses), ‘moderate’ for non-individualised homeopathy (one meta-analysis) and ‘moderate’ for all homeopathy types (three meta-analyses).
After restricting the sources of evidence to the three meta-analyses with low risk of bias (Q 20), the confidence in the cumulative evidence for all homeopathy types was now rated as 'high', the ratings for individualised and non-individualised homeopathy remained unchanged (detailed description in Additional file 3, more on this assessment in Q 30).
Here, too, we compared with corresponding quality assessments from other systematic reviews: In an analysis of 608 Cochrane reviews on therapeutic procedures from all over medicine from 2013-2014, the confidence in the cumulative evidence for efficacy of the intervention, assessed for the primary outcome, was 'high' for only 13 percent, 'moderate' for 31 percent, 'low' for 32 percent and 'very low' for 24 percent of the interventions. Withstanding all limitations of such comparisons, the balance for homeopathy is remarkable.
Q 23. What were the main reasons for your ratings of confidence in the cumulative evidence?
Evidence from meta-analyses of randomised, placebo-controlled trials is generally considered to be the evidence with the lowest risk of bias.
For all three homeopathy types (individualised, non-individualised, all homeopathy types), the first primary analysis in all available meta-analyses showed a positive significant effect of homeopathy compared to placebo. Accordingly, the confidence in the cumulative evidence was initially rated as high.
Q 24. Could the inclusion of the same trials in several meta-analyses have distorted the results of your systematic review?
No. The main result of our review (Q 21) was the proportion of meta-analyses with significantly positive effects of homeopathy over placebo, i.e. a quotient, not a sum or multiplication. Thus, using the same trials several times does not result in any additive or multiplicative magnification of effects, which would have meant a bias towards larger effects.
Q 25. But if the trials that were included in several meta-analyses had better results (more often statistically significant positive effects) by chance, couldn’t that lead to false-positive overall results?
That could be possible, but the trials included in several meta-analyses (Q 14) could also have worse results than the trials included only once. After all, the trials were not included in several meta-analyses because of their results, but because they met the respective eligibility criteria and had been published at the respective time. Accordingly, the proportion of trials with a significant positive homeopathy effect beyond placebo was
The difference of 0.2% is irrelevant and statistically non-significant (p = 0.9444); there was therefore no bias.
Q 26. Can the positive results be explained by context and attention effects?
No. In all trials of the six meta-analyses included in our systematic review, the control patients received a placebo. Accordingly, context and attention effects, observation bias, etc. should have the same impact in intervention and control groups, while any additional effects can be attributed to the intervention, namely the homeopathic remedy.
Q 27. Are there any other limitations to your systematic review?
The following limitations apply:
Q 28. What were the most surprising findings for you when working on the review?
We were surprised by the comparisons regarding the methodological quality of the homeopathy trials (Q 19) and the confidence in cumulative evidence for efficacy of homeopathy (Q 22) with other trials and reviews, respectively. Quality deficiencies of homeopathy trials have often been commented on in the literature. The comparisons, however, showed the quality of homeopathy trials to be at least as good as for other trials; the same was the case for the confidence of the cumulative evidence for efficacy of homeopathy.
Another striking finding was the completely divergent risk of bias ratings of the six meta-analyses. Of three possible rating categories, three of the meta-analyses received the best rating, three the worst, and no meta-analysis received the middle rating. Particularly striking was the top ranking — not only of the two most recent meta-analyses from 2014 and 2017 but also of the oldest from 1997 (Q 20).
Conclusions
Q 29. What conclusions did you draw from your systematic review?
According to this systematic review, homeopathy can have positive effects beyond placebo in sick people. Notably, this systematic review is based on the best available evidence — meta-analyses of randomised, placebo-controlled trials — and was conducted and published in accordance with contemporary standards.
This new level of knowledge does not legitimize measures against homeopathy in health care.
Regarding further research, the implications of the systematic review differed with regard to the different homeopathy types:
Q 30. To what extent does this systematic review come to different assessments of the included clinical trials than the authors of previous meta-analyses? How can this be explained?
Compared to the conclusions in the six meta-analyses, our conclusions are more clearly positive regarding the efficacy of homeopathy over placebo. There are several reasons for this:
What happens next?
Q 31. Are there any relevant new clinical trials on homeopathy that are not included in previous meta-analyses?
Absolutely. The HOMIS study database at the Institute for Complementary and Integrative Medicine of the University of Berne contains information on randomised and other controlled clinical homeopathy studies. A new meta-analysis would be useful for individualised homeopathy (see Q 29). For the period 2014-2021 (since the last meta-analysis on this topic by Mathie 2014), HOMIS lists an increase in 26 randomised, placebo-controlled trials on individualised homeopathy, 23 of which were published in peer-reviewed journals. How many of these would actually meet all eligibility criteria for a future systematic review and have results compatible with meta-analysis remains to be seen.
© Creative Commons CC BY-ND 4.0.
https://creativecommons.org/licenses/by-nd/4.0/
Weiter: 4.3 Homöopathie und Materiebegriff
Zurück: FORSCHUNG - Übersicht
Wir benötigen Ihre Zustimmung zum Laden der Übersetzungen
Wir nutzen einen Drittanbieter-Service, um den Inhalt der Website zu übersetzen, der möglicherweise Daten über Ihre Aktivitäten sammelt. Bitte überprüfen Sie die Details in der Datenschutzerklärung und akzeptieren Sie den Dienst, um die Übersetzungen zu sehen.