Bayesian model-averaged meta-analysis allows researchers to seamlessly incorporate available prior information into the analysis (Bartoš, Gronau, et al., 2021; Gronau et al., 2017, 2021). This vignette illustrates how to do this with an example from Bartoš, Gronau, et al. (2021), who developed informed prior distributions for meta-analyses of continuous outcomes based on the Cochrane database of systematic reviews. Then, we extend the example by incorporating publication bias adjustment with robust Bayesian meta-analysis (Bartoš, Maier, et al., 2021; Maier et al., in press).

We illustrate how to fit the informed BMA (not adjusting for publication bias) using the `RoBMA`

R package. For this purpose, we reproduce the dentine hypersensitivity example from Bartoš, Gronau, et al. (2021), who reanalyzed five studies with a tactile outcome assessment that were subjected to a meta-analysis by Poulsen et al. (2006).

We load the dentine hypersensitivity data included in the package.

```
library(RoBMA)
data("Poulsen2006", package = "RoBMA")
Poulsen2006#> d se study
#> 1 0.9073050 0.2720456 STD-Schiff-1994
#> 2 0.7207589 0.1977769 STD-Silverman-1996
#> 3 1.3305829 0.2721648 STD-Sowinski-2000
#> 4 1.7688872 0.2656483 STD-Schiff-2000-(2)
#> 5 1.3286828 0.3582617 STD-Schiff-1998
```

To reproduce the analysis from the example, we need to set informed empirical prior distributions for the effect sizes (\(\mu\)) and heterogeneity (\(\tau\)) parameters that Bartoš, Gronau, et al. (2021) obtained from the Cochrane database of systematic reviews. We can either set them manually,

```
<- RoBMA(d = Poulsen2006$d, se = Poulsen2006$se, study_names = Poulsen2006$study,
fit_BMA priors_effect = prior(distribution = "t", parameters = list(location = 0, scale = 0.51, df = 5)),
priors_heterogeneity = prior(distribution = "invgamma", parameters = list(shape = 1.79, scale = 0.28)),
priors_bias = NULL,
transformation = "cohens_d", seed = 1, parallel = TRUE)
```

with `priors_effect`

and `priors_heterogeneity`

corresponding to the \(\delta \sim T(0,0.51,5)\) and \(\tau \sim InvGamma(1.79,0.28)\) informed prior distributions for the “oral health” subfield and removing the publication bias adjustment models by setting `priors_bias = NULL`

\(^1\).

Alternatively, we can utilize the `informed_prior`

function that prepares informed prior distributions for the individual medical subfields automatically.

```
<- RoBMA(d = Poulsen2006$d, se = Poulsen2006$se, study_names = Poulsen2006$study,
fit_BMA priors_effect = prior_informed(name = "oral health", parameter = "effect", type = "smd"),
priors_heterogeneity = prior_informed(name = "oral health", parameter = "heterogeneity", type = "smd"),
priors_bias = NULL,
transformation = "cohens_d", seed = 1, parallel = TRUE)
```

The `name`

argument specifies the medical subfield name (use `print(BayesTools::prior_informed_medicine_names)`

to check names of all available subfields). The `parameter`

argument specifies whether we want prior distribution for the effect size or heterogeneity. Finally, the `type`

argument specifies what type of measure we use for the meta-analysis (only standardized mean differences `smd`

are available at the time of writing the vignette; see `?prior_informed`

for more details regarding the informed prior distributions).

We obtain the output with the `summary`

function. Adding the `conditional = TRUE`

argument allows us to inspect the conditional estimates, i.e., the effect size estimate assuming that the models specifying the presence of the effect are true and the heterogeneity estimates assuming that the models specifying the presence of heterogeneity are true\(^2\).

```
summary(fit_BMA, conditional = TRUE)
#> Call:
#> RoBMA(d = Poulsen2006$d, se = Poulsen2006$se, study_names = Poulsen2006$study,
#> transformation = "cohens_d", priors_effect = prior_informed(name = "oral health",
#> parameter = "effect", type = "smd"), priors_heterogeneity = prior_informed(name = "oral health",
#> parameter = "heterogeneity", type = "smd"), priors_bias = NULL,
#> parallel = TRUE, seed = 1)
#>
#> Robust Bayesian meta-analysis
#> Components summary:
#> Models Prior prob. Post. prob. Inclusion BF
#> 1 2/4 0.500 0.995 217.517
#> 2 2/4 0.500 0.778 3.511
#> 3 0/4 0.000 0.000 0.000
#>
#> Model-averaged estimates:
#> Mean Median 0.025 0.975
#> mu 1.076 1.088 0.664 1.422
#> tau 0.231 0.208 0.000 0.726
#> The estimates are summarized on the Cohen's d scale (priors were specified on the Cohen's d scale).
#>
#> Conditional estimates:
#> Mean Median 0.025 0.975
#> mu 1.082 1.090 0.701 1.422
#> tau 0.297 0.255 0.075 0.779
#> The estimates are summarized on the Cohen's d scale (priors were specified on the Cohen's d scale).
```

The output from the `summary.RoBMA()`

function has 3 parts. The first one under the ‘Robust Bayesian Meta-Analysis’ heading provides a basic summary of the fitted models by component types (presence of the Effect/Heterogeneity/Publication bias). We can see that there are no models correcting for publication bias (we disabled them by setting `priors_bias = NULL`

). Furthermore, the table summarizes the prior and posterior probabilities and the inclusion Bayes factors of the individual components. The results show that the inclusion Bayes factor for the effect corresponds to the one reported in Bartoš, Gronau, et al. (2021) package, \(\text{BF}_{10} = 218.53\) and \(\text{BF}_{\text{rf}} = 3.52\) (up to an MCMC error).

The second part under the ‘Model-averaged estimates’ heading displays the parameter estimates model-averaged across all specified models (i.e., including models specifying the effect size to be zero). We ignore this section and move to the last part.

The third part under the ‘Conditional estimates’ heading displays the conditional effect size estimate corresponding to the one reported in Bartoš, Gronau, et al. (2021), \(\delta = 1.082\), 95% CI [0.686,1.412], and a heterogeneity estimate (not reported previously).

The `RoBMA`

package provides extensive options for visualizing the results. Here, we visualize the prior (grey) and posterior (black) distribution for the mean parameter.

`plot(fit_BMA, parameter = "mu", prior = TRUE)`

By default, the function plots the model-averaged estimates across all models; the arrows represent the probability of a spike, and the lines represent the posterior density under models assuming non-zero effect. The secondary y-axis (right) shows the probability of the spike (at value 0) decreasing from .50, to 0.005 (also obtainable from the ‘Robust Bayesian Meta-Analysis’ field in `summary.RoBMA()`

function).

To visualize the conditional effect size estimate, we can add the `conditional = TRUE`

argument,

`plot(fit_BMA, parameter = "mu", prior = TRUE, conditional = TRUE)`

which displays only the model-averaged posterior distribution of the effect size parameter for models assuming the presence of the effect.

We can also visualize the estimates from the individual models used in the ensemble. We do that with the `plot_models()`

function, which visualizes the effect size estimates and 95% CI of each specified model included in the ensemble. Model 1 corresponds to the fixed effect model assuming the absence of the effect, \(H_0^{\text{f}}\), Model 2 corresponds to the random effect model assuming the absence of the effect, \(H_0^{\text{r}}\), Model 3 corresponds to the fixed effect model assuming the presence of the effect, \(H_3^{\text{f}}\), and Model 4 corresponds to the random effect model assuming the presence of the effect, \(H_1^{\text{r}}\),). The size of the square representing the mean estimate reflects the posterior model probability of the model, which is also displayed in the right-hand side panel. The bottom part of the figure shows the model averaged-estimate that is a combination of the individual model posterior distributions weighted by the posterior model probabilities.

`plot_models(fit_BMA)`

We see that the posterior model probability of the first two models decreased to essentially zero (when rounding to two decimals), completely omitting their estimates from the figure. Furthermore, the much larger box of Model 4 (the random effect model assuming the presence of the effect) shows that Model 4 received the largest share of the posterior probability, \(P(H_1^{\text{r}}) = 0.77\))

The last type of visualization that we show here is the forest plot. It displays the original studies’ effects and the meta-analytic estimate within one figure. It can be requested by using the `forest()`

function. Here, we again set the `conditional = TRUE`

argument to display the conditional model-averaged effect size estimate at the bottom.

`forest(fit_BMA, conditional = TRUE)`

For more options provided by the plotting function, see its documentation using `?plot.RoBMA()`

, `?plot_models()`

, and `?forest()`

.

Finally, we illustrate how to adjust our informed BMA for publication bias with robust Bayesian meta-analysis Maier et al. (in press). In short, we specify additional models assuming the presence of the publication bias and correcting for it by either specifying a selection model operating on \(p\)-values (Vevea & Hedges, 1995) or by specifying a publication bias adjustment method correcting for the relationship between effect sizes and standard errors – PET-PEESE (Stanley, 2017; Stanley & Doucouliagos, 2014). See Bartoš et al. (in press) for a tutorial.

To obtain a proper before and after publication bias adjustment comparison, we fit the informed BMA model but using the default effect size transformation (Fisher’s \(z\)).

```
<- RoBMA(d = Poulsen2006$d, se = Poulsen2006$se, study_names = Poulsen2006$study,
fit_BMAb priors_effect = prior_informed(name = "oral health", parameter = "effect", type = "smd"),
priors_heterogeneity = prior_informed(name = "oral health", parameter = "heterogeneity", type = "smd"),
priors_bias = NULL,
seed = 1, parallel = TRUE)
```

```
summary(fit_BMAb, conditional = TRUE)
#> Call:
#> RoBMA(d = Poulsen2006$d, se = Poulsen2006$se, study_names = Poulsen2006$study,
#> priors_effect = prior_informed(name = "oral health", parameter = "effect",
#> type = "smd"), priors_heterogeneity = prior_informed(name = "oral health",
#> parameter = "heterogeneity", type = "smd"), priors_bias = NULL,
#> parallel = TRUE, seed = 1)
#>
#> Robust Bayesian meta-analysis
#> Components summary:
#> Models Prior prob. Post. prob. Inclusion BF
#> 1 2/4 0.500 0.997 347.932
#> 2 2/4 0.500 0.723 2.608
#> 3 0/4 0.000 0.000 0.000
#>
#> Model-averaged estimates:
#> Mean Median 0.025 0.975
#> mu 1.045 1.052 0.705 1.344
#> tau 0.186 0.163 0.000 0.623
#> The estimates are summarized on the Cohen's d scale (priors were specified on the Cohen's d scale).
#>
#> Conditional estimates:
#> Mean Median 0.025 0.975
#> mu 1.048 1.053 0.720 1.344
#> tau 0.256 0.220 0.064 0.681
#> The estimates are summarized on the Cohen's d scale (priors were specified on the Cohen's d scale).
```

We obtain noticeably stronger evidence for the presence of the effect. This is a result of placing more weights on the fixed-effect models, especially the fixed-effect model assuming the presence of the effect \(H_1^f\). The increase in the posterior model probability of \(H_1^f\) is, in our case, the fact that it predicted the data slightly better after removing the correlation between effect sizes and standard errors (a result of using Cohen’s \(d\) for model fitting). Nevertheless, the conditional effect size estimate stayed almost the same.

Now, we fit the publication bias-adjusted model by simply removing the `priors_bias = NULL`

, which allows us to obtain the default 36 models ensemble called RoBMA-PSMA (Bartoš, Maier, et al., 2021).

```
<- RoBMA(d = Poulsen2006$d, se = Poulsen2006$se, study_names = Poulsen2006$study,
fit_RoBMA priors_effect = prior_informed(name = "oral health", parameter = "effect", type = "smd"),
priors_heterogeneity = prior_informed(name = "oral health", parameter = "heterogeneity", type = "smd"),
seed = 1, parallel = TRUE)
```

```
summary(fit_RoBMA, conditional = TRUE)
#> Call:
#> RoBMA(d = Poulsen2006$d, se = Poulsen2006$se, study_names = Poulsen2006$study,
#> priors_effect = prior_informed(name = "oral health", parameter = "effect",
#> type = "smd"), priors_heterogeneity = prior_informed(name = "oral health",
#> parameter = "heterogeneity", type = "smd"), parallel = TRUE,
#> seed = 1)
#>
#> Robust Bayesian meta-analysis
#> Components summary:
#> Models Prior prob. Post. prob. Inclusion BF
#> 1 18/36 0.500 0.858 6.022
#> 2 18/36 0.500 0.714 2.502
#> 3 32/36 0.500 0.697 2.304
#>
#> Model-averaged estimates:
#> Mean Median 0.025 0.975
#> mu 0.722 0.880 0.000 1.283
#> tau 0.202 0.161 0.000 0.799
#> omega[0,0.025] 1.000 1.000 1.000 1.000
#> omega[0.025,0.05] 0.943 1.000 0.329 1.000
#> omega[0.05,0.5] 0.874 1.000 0.071 1.000
#> omega[0.5,0.95] 0.855 1.000 0.042 1.000
#> omega[0.95,0.975] 0.866 1.000 0.050 1.000
#> omega[0.975,1] 0.897 1.000 0.057 1.000
#> PET 0.931 0.000 0.000 4.927
#> PEESE 1.131 0.000 0.000 12.261
#> The estimates are summarized on the Cohen's d scale (priors were specified on the Cohen's d scale).
#> (Estimated publication weights omega correspond to one-sided p-values.)
#>
#> Conditional estimates:
#> Mean Median 0.025 0.975
#> mu 0.838 0.938 -0.035 1.297
#> tau 0.285 0.227 0.064 0.906
#> omega[0,0.025] 1.000 1.000 1.000 1.000
#> omega[0.025,0.05] 0.736 0.829 0.092 1.000
#> omega[0.05,0.5] 0.411 0.373 0.014 0.951
#> omega[0.5,0.95] 0.320 0.249 0.008 0.919
#> omega[0.95,0.975] 0.376 0.311 0.009 0.958
#> omega[0.975,1] 0.518 0.425 0.010 1.000
#> PET 2.909 3.136 0.171 5.614
#> PEESE 7.048 6.034 0.375 18.162
#> The estimates are summarized on the Cohen's d scale (priors were specified on the Cohen's d scale).
#> (Estimated publication weights omega correspond to one-sided p-values.)
```

We notice that additional values in the ‘Components summary’ table in the ‘Bias’ row. The model is now extended with 32 publication bias adjustment models that account for 50% of the prior model probability. When comparing the RoBMA to the second BMA fit, we notice a large decrease in the inclusion Bayes factor for the presence of the effect \(\text{BF}_{10} = 6.02\) vs. \(\text{BF}_{10} = 347.93\), which still, however, presents moderate evidence for the presence of the effect. We can further quantify the evidence in favor of the publication bias with the inclusion Bayes factor for publication bias \(\text{BF}_{pb} = 2.30\), which can be interpreted as weak evidence in favor of the publication bias.

We can also compare the publication bias unadjusted and publication bias-adjusted conditional effect size estimates. Including models assuming publication bias into our model-averaged estimate (assuming the presence of the effect) slightly decreases the estimated effect to \(\delta = 0.838\), 95% CI [-0.035, 1.297] with a much wider confidence interval, as visualized in the prior and posterior conditional effect size estimate plot.

`plot(fit_RoBMA, parameter = "mu", prior = TRUE, conditional = TRUE)`

\(^1\) The additional setting `transformation = "cohens_d"`

allows us to get more comparable results with the `metaBMA`

R package since RoBMA otherwise internally transforms the effect sizes to Fisher’s \(z\) for the fitting purposes and the `seed = 1`

and `parallel = TRUE`

options grant us exact reproducibility of the results and parallelization of the fitting process.

\(^2\) The Model-averaged estimates that `RoBMA`

returns by default model-averaged across all specified models – a different behavior from the `metaBMA`

package that by default returns what we call “conditional” estimates in `RoBMA`

.

Bartoš, F., Gronau, Q. F., Timmers, B., Otte, W. M., Ly, A., & Wagenmakers, E.-J. (2021). Bayesian model-averaged meta-analysis in medicine. *Statistics in Medicine*. https://doi.org/10.1002/sim.9170

Bartoš, F., Maier, M., Quintana, D. S., & Wagenmakers, E.-J. (in press). Adjusting for publication bias in JASP & R – selection models, PET-PEESE, and robust Bayesian meta-analysis. *Advances in Methods and Practices in Psychological Science*. https://doi.org/10.31234/osf.io/75bqn

Bartoš, F., Maier, M., Wagenmakers, E.-J., Doucouliagos, H., & Stanley, T. D. (2021). *Need to choose: Robust Bayesian meta-analysis with competing publication bias adjustment methods*. PsyArXiv. https://doi.org/10.31234/osf.io/kvsp7

Gronau, Q. F., Heck, D. W., Berkhout, S. W., Haaf, J. M., & Wagenmakers, E.-J. (2021). A primer on Bayesian model-averaged meta-analysis. *Advances in Methods and Practices in Psychological Science*, *4*(3), 25152459211031256. https://doi.org/10.1177/25152459211031256

Gronau, Q. F., Van Erp, S., Heck, D. W., Cesario, J., Jonas, K. J., & Wagenmakers, E.-J. (2017). A Bayesian model-averaged meta-analysis of the power pose effect with informed and default priors: The case of felt power. *Comprehensive Results in Social Psychology*, *2*(1), 123–138. https://doi.org/10.1080/23743603.2017.1326760

Maier, M., Bartoš, F., & Wagenmakers, E.-J. (in press). Robust Bayesian meta-analysis: Addressing publication bias with model-averaging. *Psychological Methods*. https://doi.org/10.31234/osf.io/u4cns

Poulsen, S., Errboe, M., Mevil, Y. L., & Glenny, A.-M. (2006). Potassium containing toothpastes for dentine hypersensitivity. *Cochrane Database of Systematic Reviews*, *3*. https://doi.org/10.1002/14651858.cd001476.pub2

Stanley, T. D. (2017). Limitations of PET-PEESE and other meta-analysis methods. *Social Psychological and Personality Science*, *8*(5), 581–591. https://doi.org/10.1177/1948550617693062

Stanley, T. D., & Doucouliagos, H. (2014). Meta-regression approximations to reduce publication selection bias. *Research Synthesis Methods*, *5*(1), 60–78. https://doi.org/10.1002/jrsm.1095

Vevea, J. L., & Hedges, L. V. (1995). A general linear model for estimating effect size in the presence of publication bias. *Psychometrika*, *60*(3), 419–435. https://doi.org/10.1007/BF02294384