1 Introduction

Publication bias (or the file-drawer effect) poses undoubtedly a serious threat to the integrity of research. However, researchers hold divergent views on its prevalence, ranging from the dramatic claim that “most published research findings are false” (see Goodman and Greenland (2007) for commentary), to the argument that “file-drawer sizes are likely much smaller, and Questionable Research Practices less abundant, than required by false positive models” (Bak-Coleman et al. 2022).

The primary aim of research on publication bias is to mitigate its effects, that is, to correct the evidential value of study sets, particularly in meta-analyses. This approach can lead to meaningful findings (see, for example, Maier et al. 2022) and is a recommended practice in meta-analyses (Deeks et al. 2019). A secondary goal, which we might term meta-scientific, is to examine the prevalence, impact, and underlying causes of publication bias itself, independent of its effects on specific findings.

These two objectives are similar but distinct, and they share some common methodological challenges. Notably, no method exists that can be described as simultaneously well-known, popular, and state-of-the-art for addressing both purposes. While many methods are available for correcting evidential value (see: Furuya-Kanamori et al. 2020; Renkewitz and Keiner 2019; Bartoš, Maier, et al. 2023), they often struggle when effect size vary across studies. For the meta-scientific context, or in cases of significant heterogeneity, only two existing methods warrant mention. The first is the caliper test (Gerber and Malhotra 2008), a straightforward test assessing the continuity of a probability density function around a critical value (e.g., p = 0.05). The second, more sophisticated method, is the Z-Curve, originally proposed by Brunner and Schimmack (2020) and further refined by Bartoš and Schimmack (2022).

Both the Z-Curve and caliper methods have their shortcomings, omitting significant amounts of information, leaving substantial room for improvement. The field of meta-science would benefit from more rigorous methods, as the lack of proper rigor is cited as a limitation, particularly when meta-scientists draw conclusions about policy (Devezer et al. 2021). Such rigor is harder to obtain using conceptually limited methods, like in study made by Brodeur et al. (2024) or analyses conducted under the assumption of no bias like in van Zwet et al. (2023) analysis of clinical trials.

The aim of the work behind this article is to develop a new method for studying publication bias in a meta-scientific context, with potential applications in meta-analytical contexts as well. The concept for this new test stems from previous work where the Z-Curve was initially applied Plisiecki and Lenartowicz 2023; Plisiecki, Lenartowicz, et al. 2022. The proposed method builds upon the Z-Curve approach by utilizing all available statistical results for estimation. In addition to estimating bias, it introduces the capability to compare the model to the null hypothesis of no bias, an option that is only indirectly available with the Z-Curve and absent in the caliper test.

2 Idea

The Likelihood Ratio Test for Publication Bias operates on sets of p-values, representing the results of statistical tests. The test compares the likelihood ratio of the null hypothesis, which assumes no biased studies, against the alternative hypothesis, which posits the presence of bias:

λ_LR = −2 [l(θ₀) − l(θ₁)] ,

where l(θ₁) corresponds to the logarithm of the (maximized) likelihood function of the alternative hypothesis that includes a biased component, and l(θ₀) corresponds to the null hypothesis – a model with no bias.

This concept is versatile, making it suitable both for meta-analyses, which examine datasets from studies focusing on single effects, and for datasets with more heterogeneous sources, such as those derived from text mining, where a prominent methodological gap exists. Since the primary objective of this work is to address the latter gap, the following discussion will focus on the implementation developed to meet this need. It’s also worth noting that the idea allows other approach than maximized likelihood, which is an interesting idea to be explored in the future.

To specify the possible parameter space Θ : θ₀, θ₁ ∈ Θ, this test utilizes conversion from p-values to z-score and approximate by mixture model of accordingly modified (Gaussian distribution. In the proposed approach, the parameter space of the null hypothesis can be defined as π_k, that represents weight of each component (so: ) and k represents non-centrality parameter of the density function:

Where f_Folded(x | µ_k) = ϕ(x | µ_k) + ϕ(−x | µ_k), and K is a number of components. For alternative hypothesis, to K non-biased components we add B biased components:

The most straightforward choice for f_Biased is a Gaussian distribution that is left-truncated at a specified threshold, such as c > 1.96, which corresponds to p = 0.05. Representing the idea that some test results are published only if the test itself is statistically significant at a common threshold.

In this setup, l(θ₀) represents a constrained model of l(θ₁), where the weight of biased components is set to zero. That helps us in establishing statistical inference with specified levels of significance (α) and power (β), providing a robust framework for detecting publication bias.  Also, simulation results are promising in supporting the applicability of Wilks’ theorem to this test, suggesting that it could be a viable method for rapid p-value calculation. However, determining the correct number of degrees of freedom is crucial and challenging analytically, so simulations will be a critical tool for this method, both for verifying assumptions about the chi-square distribution and for providing an alternative way for obtaining p-value for null hypothesis.

Approximation by the folded normal distribution is justified by the property of the z-score distribution of a single effect, such that z-scored derived from p-values should follow (non-central) truncated Gaussian distribution. This is a trivial and common approach for estimating the p-value distribution of z-scores under the true null hypothesis. Additionally, several meta-scientific studies have employed this property for specific families of tests, such as t-tests, under the true alternative hypothesis, where the distribution of z-scores is shifted (see Thomas 1997; Elliott, Kudrin, and Wüthrich 2022). While some studies apply this assumption across all tests (see Brunner and Schimmack 2020; (see Bartoš and Schimmack 2022; Hoenig and Heisey 2001; Cumming 2008; Hedges 1984), the theoretical justification is not trivial.

While it’s achievable to show, that for the vast majority of tests, based on t or chi-squared distribution, p-values translated into z-scores, follows normal distribution, at least asymptotically, the real problem is robustness. It is especially because such property is much more vulnerable to violation of tests assumptions than regular null hypothesis testing². On the other hand, this assumption is less crucial in our proposed method because we use it to approximate z-scores with only a few components, focusing on differences around critical values by comparing two such mixed distributions. This approach has been successfully applied in Z-Curve methods (Brunner and Schimmack 2020; Bartoš and Schimmack 2022). Still, a comprehensive test of the robustness of this approximation method is needed and, to my knowledge, has not yet been conducted, even for similar setups for other methods. For further considerations, see the Limitations section.

3 Implementation and results

As a proof of concept, I developed an implementation in python, that could be used in the future as a statistical package for the testing of publication bias. The truncating threshold is set at c = 1.96. For the unbiased component, there are 6 folded normal distributions, one of which has a pre-mean fixed at 0 to represent true null results. The biased component uses six truncated distributions with pre-truncation mean of 0, 1, 1.96, 3, 4 and 5. For this purpose, I developed a python implementation of the Expectation–maximization (EM) algorithm, to compute the maximized likelihood function. Algorithm, as code prepared to create a package, along with presented here simulations, are available at GitHub.³

3.1 Simulation data

For the simulation, synthetic data in a 4×4×4×20 design was created. The first factor represents the number of different true effects in the dataset: 1, 4, 9, or 16 different true effects, where each of them is represented by a different folded normal distribution with a non-centrality parameter between 0.1 and 4. For each of the 4 sets of factors, 10,000 z-scores were generated. The second factor represents the proportion of 0.2, 0.4, 0.6, or 0.8 that are true null components in each dataset.

For each 4 × 4 factor set and each of the 20 bias levels (ranging from 0% to 95% in 5% increments), a combined dataset was created by merging true and null results. To simulate bias, non-significant results (z < 1.96) were removed according to the specified bias percentage. Then, 100 samples were drawn for each sample size of 100, 200, 400, and 800 z-scores.

This design yielded:

• 6,400 unbiased samples (4 true effects × 4 null proportions × 4 sample sizes × 100 replications with a bias level of 0%).
• 121,600 biased samples (4 true effects × 4 null proportions × 4 sample sizes × 19 bias levels × 100 replications with bias).

The purpose of this design is to create diverse datasets in terms of the distribution of true effects, also, this approach allows for scenarios where the true number of components both exceeds and falls short of the number of simulated components. While this design does not exhaust all possible methods for generating such data, additional designs should be simulated and emphasize varying assumptions about the non-uniform probability of publication and include controlled departures from assumptions about test correctness and independence. For more technical details, see the attached code.

3.2 Type I error

To evaluate the robustness and α-level properties of the test, we computed the difference in log-likelihood for each of the 6,400 unbiased samples and compared these differences to a chi-squared distribution with fitted degrees of freedom set to 0.55. The results, illustrated in Plot 1, are encouraging and indicate a reasonable fit. Specifically, when the degrees of freedom were aligned to the critical values, 5.36% of the observed values exceeded the theoretical 5% threshold, and 1.00% of the observed values exceeded the 1% threshold.

Figure 1: QQ plot of test results against a chi-squared distribution with df = 0.55.

Since the degrees of freedom likely cannot be derived analytically (as is the case in much simpler statistical settings) and depend on several parameters of the EM algorithm, it is recommended to use the default values or recalibrate the degrees of freedom as needed. The degrees of freedom are robust to changes in sample size (as shown in Figure 1) and other bias parameters. Negative values of the test statistic are an artifact of the EM algorithm, and do not affect the results.

3.3     Type II error

For the purpose of illustrating the ability to detect bias and the asymptotically increasing power as both bias and sample sizes grow, analyses were conducted on each of the 121600 samples. The tests were grouped by sample size, and the relationship between the estimated and true percentage of unpublished results was examined (Figure 2).

Figure 2: Correlation between true and estimated value of unreported results for each sample size.

Figure 3: Power of the test for combinations of sample size and bias magnitude.

A clear correlation between the true and estimated bias ratios is visible, along with increasing power and precision as sample sizes grow. Furthermore, there are no indications of underestimation or overestimation bias in this method. However, the results are contingent on the bias characteristics in the generated data, so no strong conclusions can be drawn (see the Limitations section). Additionally, power across different sample sizes and bias degrees in the dataset is presented in Figure 3.

3.4 Confidence intervals

4 Comparison with other methods

Figure 4: Comparison of the LRBT, Caliper Test, original Z-Curve2, and modified Z-Curve New Method. MSE – mean squared error of estimation, Bias – mean difference between estimated and true value, Std Dev – standard deviation of estimated value, Correlation – Spearman correlation between estimated and true value

5 Limitations

Publication bias tests face inherent limitations that cannot be fully resolved through methodological approaches alone. The primary challenge lies in the difficulty of defining the specific characteristics of publication probability a priori. Bias can manifest at various thresholds and may be associated with different probabilities of publication, influenced by factors beyond the statistical test results themselves. Consequently, the accuracy of any bias estimation will ultimately depend on the validity of these underlying assumptions.

The Likelihood Ratio Test for Publication Bias introduced here assumes that certain components represent studies published only when their effect size exceeds a critical value and reaches statistical significance. This is equivalent to assuming that the probability of publication is roughly uniform across all non-significant results. Deviations from this assumption, such as more nuanced forms of selective censoring, may lead to discrepancies between estimated and true values.

There are several issues arising from the approximation via Gaussian distribution. Certain test results may not follow this distribution, so this assumption should be checked before applying this test (as with most other bias tests). As mentioned, a robustness analysis of this assumption is planned.

Another issue, which to my knowledge has not been widely discussed, is that p-curves — and consequently the z-values of some tests based on the t-distribution — do not always follow their theoretical distribution. This discrepancy arises when a t-test is applied to a heavy-tailed distribution, where individual data points can disproportionately impact the standard deviation estimate. In such cases, the distribution of t-statistics can shift toward values around 1 or -1. This characteristic, while making the t-test robust and conservative under assumption violations by preventing inflation of the error rate beyond the nominal α-level (at the cost of statistical power), complicates interpretation near the significance threshold.

Discrepancies can also arise with tests producing discrete results, such as rank-based tests or edge cases like testing correlation when the true r is 1 or -1. Additionally, issues with rounding p-values may occur. Although unrounding has not been implemented yet in this method, it is an established approach that could help address some limitations in p-curve and z-curve analyses (Elliott, Kudrin, and Wüthrich 2022). In datasets derived from text mining, there is an added risk that some results may be drawn from overlapping data sources (Bishop and Thompson 2016), leading to correlated outcomes. Additionally, some tests within these datasets may have been conducted sequentially, introducing dependency structures that are often unaccounted for. These issues are generally less problematic when datasets are sourced from diverse, independent studies, as a variety of sources reduces the overlap and dependency.

Text-mined datasets may also exhibit non-existing bias due to differences in publication formats, which can influence how significant and non-significant results are reported.  While guidelines like APA style aim to standardize reporting, variations in publication practices can lead to inconsistencies. For example, significant results may be emphasized or formatted differently from non-significant ones, complicating efforts to maintain uniformity.

Another significant challenge is the impact of practices that inflate false positive rates or involve fraud. Such practices can artificially increase effect sizes and create misleading patterns, particularly but not exclusively around statistical significance thresholds. While some of these practices may intuitively seem to increase the estimated degree of publication bias — a beneficial outcome for bias-detection methods their actual impact is difficult to predict and likely impossible to quantify accurately.

Also, as the simulations show, detecting publication bias in samples of a couple of hundreds or less p-values is non-reliable, unless the degree of bias is spectacular, which is however, sadly, often the case. That need for considerably big samples limits the potential applications of bias detection method, and it isn’t a new concern, at least for estimation of publication bias in meta-analysis (Furuya-Kanamori et al. 2020, Renkewitz and Keiner 2019).

6 Conclusion

The Likelihood Ratio Test for Publication Bias presented here offers a significant advancement in detecting and quantifying publication bias in datasets. This framework enables statistical inference with specified levels of significance (α) and power (β) and facilitates the estimation and comparison of publication bias magnitudes. Although this study places strong emphasis on the test’s limitations, the assumptions required for this test are no more restrictive than those in methods commonly used in current research.

Initial simulations support the applicability of Wilks’ theorem to this approach, indicating its potential for robust and fast p-value calculation, which is a minor advancement over existing methods. Comparisons with existing methods, such as Z-Curve 2 and the Caliper test, show that the Likelihood Ratio Bias Test performs better across key metrics, including mean squared error, estimation bias, and Spearman correlation. This suggests that it is a much more robust and powerful tool for detecting and quantifying publication bias in datasets. Moreover, the current Python implementation, although effective, has identified minor algorithmic issues, and improvements in highlighted areas could enhance the method’s performance in future applications.

Nonetheless, inherent challenges in publication bias estimation persist, and two strong recommendations for reliable use are proposed:

1. Unless careful justification is provided, analysis for publication bias should be conducted on datasets containing at least several hundred, preferably over 1,000 test results, that come from diverse sources. This should help reduce the risks from autocorrelation, deviations from test assumptions, and the risk of conducting underpowered Exceptions to this recommendation could include carefully selected, so-called “focal hypotheses” or meta-analyses, but only when a large publication bias effect is expected.

2. When comparing publication bias between different datasets, ensure that the two datasets were produced using a similar methodology, both in terms of publication culture, time period, and the method of acquiring the test results (e.g., text mining).

Notes

2. An example of a problem may be that some tests react to breaking assumptions by being conservative, which is less of a problem for tests but huge for some meta-analytical methods. Also, technically speaking, assumption.
3. https://github.com/pawlenartowicz/LikelihoodRatioBiasTest

References

Bak-Coleman, Joseph et al. (Apr. 28, 2022). Revisiting the Replication Crisis without False Positives. doi: 10.31235/osf.io/rkyf7. Pre-published.

Bartoš, František, Maximilian Maier, et al. (2023). “Robust Bayesian Meta-Analysis: Model-averaging across Complementary Publication Bias Adjustment Methods”. In: Research Synthesis Methods 14.1, pp. 99–116. issn: 1759-2887. doi: 10.1002/jrsm.1594.

Bartoš, František and Ulrich Schimmack (Sept. 1, 2022). “Z-Curve 2.0: Estimating Replication Rates and Discovery Rates”. In: Meta-Psychology 6. issn: 2003-2714. doi: 10.15626/MP.2021.2720.

Bishop, Dorothy V. M. and Paul A. Thompson (Feb. 18, 2016). “Problems in Using P-Curve Analysis and Text-Mining to Detect Rate of p-Hacking and Evidential Value”. In: PeerJ 4, e1715. issn: 2167-8359. doi: 10.7717/peerj. 1715.

Brodeur, Abel et al. (Aug. 2024). “Do Preregistration and Preanalysis Plans Reduce P-Hacking and Publication Bias? Evidence from 15,992 Test Statistics and Suggestions for Improvement”. In: Journal of Political Economy Microeconomics 2.3, pp. 527–561. issn: 2832-9368. doi: 10.1086/730455.

Brunner, Jerry and Ulrich Schimmack (May 31, 2020). “Estimating Population Mean Power Under Conditions of Heterogeneity and Selection for Significance”. In: Meta-Psychology 4. issn: 2003-2714. doi: 10.15626/MP.2018.874. Cumming, Geoff (July 1, 2008). “Replication and p Intervals: p Values Predict the Future Only Vaguely, but Confidence Intervals Do Much Better”. In: Perspectives on Psychological Science 3.4, pp. 286–300. issn: 1745-6916. doi: 10.1111/j.1745-6924.2008.00079.x.

Deeks, Jonathan J et al. (2019). “Analysing Data and Undertaking Meta-Analyses”. In: Cochrane Handbook for Systematic Reviews of Interventions. John Wiley & Sons, Ltd, pp. 241–284. isbn: 978-1-119-53660-4. doi: 10. 1002/9781119536604.ch10.

Devezer, Berna et al. (Mar. 31, 2021). “The Case for Formal Methodology in Scientific Reform”. In: Royal Society Open Science 8.3, p. 200805. issn: 2054-5703. doi: 10.1098/rsos.200805. pmid: 34035933.

Elliott, Graham, Nikolay Kudrin, and Kaspar Wüthrich (2022). “Detecting P-Hacking”. In: Econometrica 90.2, pp. 887–906. issn: 1468-0262. doi: 10.3982/ECTA18583.

Furuya-Kanamori, Luis et al. (Feb. 1, 2020). “P Value–Driven Methods Were Underpowered to Detect Publication Bias: Analysis of Cochrane Review Meta-Analyses”. In: Journal of Clinical Epidemiology 118, pp. 86–92. issn: 0895-4356. doi: 10.1016/j.jclinepi.2019.11.011.

Gerber, Alan and Neil Malhotra (2008). “Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals”. In: Quarterly Journal of Political Science 3.3, pp. 313–326. doi: 10.1561/100.00008024.

Goodman, Steven and Sander Greenland (Apr. 24, 2007). “Why Most Published Research Findings Are False: Problems in the Analysis”. In: PLOS Medicine 4.4, e168. issn: 1549-1676. doi: 10.1371/journal.pmed.0040168.

Hedges, Larry V. (Mar. 1, 1984). “Estimation of Effect Size under Nonrandom Sampling: The Effects of Censoring Studies Yielding Statistically Insignificant Mean Differences”. In: Journal of Educational Statistics 9.1, pp. 61–85. issn: 0362-9791. doi: 10.3102/10769986009001061.

Hoenig, John M and Dennis M Heisey (Feb. 1, 2001). “The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis”. In: The American Statistician 55.1, pp. 19–24. issn: 0003-1305. doi: 10.1198/000313001300339897.

Maier, Maximilian et al. (Aug. 2, 2022). “No Evidence for Nudging after Adjusting for Publication Bias”. In: Proceedings of the National Academy of Sciences 119.31, e2200300119. doi: 10.1073/pnas.2200300119.

Plisiecki, Hubert and Paweł Lenartowicz (Nov. 20, 2023). “Looking for Evidence That Publish-or-Perish Reinforces Publication Bias and Gamification of Publishing Strategy”. In: doi: 10.17605/OSF.IO/JGRBF.

Plisiecki, Hubert, Paweł Lenartowicz, et al. (Dec. 1, 2022). “Estimating Replicability of Polish Psychology”. In: doi: 10.17605/OSF.IO/Z4WQK.

Renkewitz, Frank and Melanie Keiner (Oct. 2019). “How to Detect Publication Bias in Psychological Research”. In: Zeitschrift für Psychologie 227.4, pp. 261–279. issn: 2190-8370. doi: 10.1027/2151-2604/a000386.

Thomas, Len (1997). “Retrospective Power Analysis”. In: Conservation Biology 11.1, pp. 276–280. issn: 1523-1739. doi: 10.1046/j.1523-1739.1997.96102.x.

Van Zwet, Erik et al. (Dec. 26, 2023). “A New Look at P Values for Randomized Clinical Trials”. In: NEJM Evidence 3.1, EVIDoa2300003. doi: 10.1056/ EVIDoa2300003.