How journals catch statistical errors

Before sending a manuscript to reviewers, an editor may examine its abstract and methods and ask: does the design support the claim, is the reporting guideline satisfied, is the primary endpoint stated, do the headline numbers look internally plausible. Some journals describe their criteria. One editorial “top 10 reasons your manuscript may be rejected without review” ↗ puts inadequate design and incomplete methodological reporting near the top. A statistical problem visible in the abstract may prompt an editorial question before external review. We cover this screen in detail in why editors desk-reject before peer review.

Automated tools can check whether reported test statistics and p-values agree. statcheck recomputes p-values for the fully reported test results it recognizes in APA style, using the printed statistic and degrees of freedom where applicable. Formatting differences and results in tables can be missed, as its documentation ↗ explains. When it was run across eight psychology journals over 1985–2013, about half the papers had at least one inconsistent p-value and one in eight a gross one ↗. GRIM checks whether a reported mean of integer data is arithmetically possible for the stated sample size; in its founding study, about half of the analyzable articles contained at least one impossible mean ↗.

These checks are deterministic — they pass or they fail, and there is no arguing with the arithmetic. They are also blind to everything except consistency: they do not know whether the right test was chosen or the claim was earned. We break down each one in when a p-value doesn't match its test statistic and the GRIM test.

Reference lists get the same mechanical treatment: some journals verify that every cited DOI and PMID resolves — and resolves to the work actually cited. Our free reference integrity checker runs that exact check before you submit.

Many journals require a completed reporting checklist at submission and check the manuscript against it: CONSORT for randomized trials, STROBE for observational studies, TRIPOD for prediction models, PRISMA for systematic reviews. A missing flow diagram, an unaccounted-for dropout, an unstated handling of missing data — each is a concrete, citable reason to return the paper, and the published checklist gives authors a way to examine these gaps before submission. Walk your design's checklist before submission: our STROBE, CONSORT, TRIPOD, and PRISMA walkthroughs take them item by item.

A methodological reviewer goes past consistency to appropriateness: power and precision, multiplicity, missing-data handling, confounding, whether the conclusion outruns the result. And the scrutiny does not stop at acceptance. Statistical detectives audit the published record too — John Carlisle's survey of baseline data in 5,087 randomized trials ↗ found an excess of unusually similar or dissimilar baseline distributions. Such patterns warrant investigation and can have several causes, including errors or features of allocation. Errors caught at this stage can lead to corrections or retractions. We walk through a corrected paper in a worked review of corrected PREDIMED.

Before submission, compare the manuscript with its reporting checklist, recompute statistics where the printed inputs allow, and discuss methodological concerns with the research team. Reviewing before submission gives you time to reconcile discrepancies, revise the interpretation, or discuss further analysis with a statistician.

A structured pre-submission statistical review runs them together. RigorMD appraises a manuscript with two independent AI appraisers examining the design, analysis, and conclusions. A separate numerical layer recalculates statistics when the reported inputs allow. The full review also checks the design's reporting guideline item by item, positions its novelty against retrieved prior work, and returns a severity-scored report across seven domains, from numerical consistency to whether the conclusion matches the evidence. Supporting evidence can include manuscript passages, calculations, reference records, and retrieved prior work. It flags concerns for your judgment; it is not a substitute for peer review or a statistician's input on design. See a full sample report →, read how the engine works, or review pricing — the pre-submission review is $99.