Methods review and statistical checks, explained
RigorMD examines whether your study design and analysis support your conclusions. It also recalculates statistics when the manuscript supplies the required inputs. For each numerical check, this page explains what it needs, what it can show, and when it cannot run.
§01 How to read this page
The review considers confounding, clustering, repeated measurements, missing data, multiple comparisons, and whether the wording matches the evidence. These questions require reading the study's methods and results, alongside checking its numbers.
When the required values are readable in your files, RigorMD calculates the statistic and compares it with the reported value. A numerical finding shows the discrepancy and the calculation behind it. The catalog separates recomputation and arithmetic-impossibility checks, reference and integrity checks, and reporting-consistency screens. The total check count is not the number of recalculations.
§02 Recomputation & arithmetic-impossibility checks
The core. Where the reported numbers let us rebuild a statistic, we do — a p-value from a 2×2 table or a printed test statistic, the p-value an effect estimate's confidence interval implies, a two-group t-test from each arm's mean ± SD — and flag the ones that do not reconcile beyond what rounding explains. Each check requires the inputs listed below.
| Check | What it catches | What it needs | Max severity |
|---|---|---|---|
| GRIM | A reported mean that no whole-number data could produce — e.g. a 1–5 scale mean of 2.53 from n = 5. | integer-summed values + N | Moderate |
| p-value from a 2×2 table | Recomputes the p-value from the table under the test the manuscript names — Fisher's exact, Pearson χ², Yates-corrected χ², or a χ² that does not say which, read both with and without Yates' correction because statistical packages differ on the default — or under all three when it names none. A p-value attributed to any other named test, such as McNemar's, is not recomputed. Where the manuscript names Pearson's χ² and the exact printed p-value is the Yates-corrected value rather than the uncorrected one, the row is a note, not a contradiction: R applies the correction by default, and papers shorten its label to Pearson's. Where it names an uncorrected χ², the same match is a note that the value does not fit the stated method. An exact printed p-value that only Fisher's exact test reproduces, where the manuscript names χ², is likewise noted as a test the manuscript does not name. Each of these notes is moderate when the named test would make the opposite significance call and mild otherwise; a printed threshold never gets a fallback reading. A comparison the paper judges against its own stated threshold (a Bonferroni-corrected α, say) is also checked at that threshold; a flip found only there is moderate, and a flip at 0.05 keeps its grade. Otherwise a significance call that flips across the 0.05 line under every reading is the critical case, however it is printed; on a baseline-table row it caps at moderate. Other disagreements grade below it: a recomputed value that differs materially without changing the call, a contradicted threshold whose call still stands, or a result that depends on which test was used. | a reconstructable 2×2 table | up to Critical |
| Denominator vs analytic N | Subgroups that sum to more patients than the study reported analyzing — graded by the overflow as a share of N, so a rounding-scale slip is mild and a gross overflow serious. | subgroup counts + stated N | up to Serious |
| Percentage vs count / denominator | Recomputes each percentage from its own n / N and flags the ones that do not reconcile — graded by the size of the gap, so a small slip is mild and a gross mismatch serious. A percentage printed without its count, against a group size printed elsewhere, is flagged when no whole number of patients gives it, at most moderate. | a count + its denominator, or a group size printed elsewhere | up to Serious |
| SD / SEM mix-up | Two spreads for one quantity that differ by exactly √n — the arithmetic signature of a mislabeled standard error. | two spreads + N | Moderate |
| Cross-location consistency | The same quantity printed at different values across the abstract, body, and tables — past what rounding explains. | the value stated in ≥ 2 places | Moderate |
| Narrative claim vs the table it cites | A sweeping statement in the text — “all”, “almost every”, “most”, “none” — checked against the table cell that counts the same group. The quoted sentence must appear verbatim in the manuscript, and only a decisive conflict is flagged, graded by how far the table falls from what the wording implies. | a quantified claim + the table cell counting it | up to Serious |
| Distribution plausibility (Altman–Bland) | A non-negative measure whose mean − 2·SD falls below zero, so a normal summary is unlikely and a median (IQR) would fit better. | mean + SD, a non-negative quantity | Mild |
| Effect estimate vs its interval and p-value | An odds, hazard, or risk ratio (or a difference) must sit inside its own confidence interval, and the p-value that interval implies must agree with the printed p — beyond what rounding or the interval method explains. A baseline-table row caps at moderate. An odds ratio's 95% interval is also compared with the narrowest a standard logistic model allows from the events in its two groups, when the paper prints them; a narrower one is a mild note, because weighting, penalization, robust or clustered standard errors, pooled, standardized or ordinal estimates, and score (Cornfield) intervals can all give a narrower interval correctly. A comparison the paper judges against its own stated threshold (a Bonferroni-corrected α, say) is also checked at that threshold; a flip found only there is moderate, and a flip at 0.05 keeps its grade. | an estimate + its CI (+ its p, or an odds ratio's events per group) | up to Serious |
| p-value from a printed test statistic | Recomputes p from a printed t, F, χ², z, or r with its degrees of freedom and flags a printed p that disagrees — including a call that flips across the 0.05 line. A comparison the paper judges against its own stated threshold (a Bonferroni-corrected α, say) is also checked at that threshold; a flip found only there is moderate, and a flip at 0.05 keeps its grade. | a test statistic + df + p | up to Serious |
| Two-group mean ± SD comparison | Recomputes the t-test from each group's mean, SD, and n under the test the paper states, and flags a printed p that disagrees. A comparison the paper judges against its own stated threshold (a Bonferroni-corrected α, say) is also checked at that threshold; a flip found only there is moderate, and a flip at 0.05 keeps its grade. | mean ± SD + n per group + p | up to Serious |
| Category percentages sum to 100 | The levels of one categorical variable (sex, ASA class, stage) must sum to 100% within rounding, and their counts must not exceed the group N (a shortfall is missing data — reported, not flagged). | one variable's level percentages | up to Serious |
| Summary statistics within their own bounds | A median must lie inside its IQR, every summary inside the range, and all of them inside a stated scale. | median + IQR, or a range, or a scale | Moderate |
| Subset mean ± SD vs its whole group | When a paper gives a mean and SD for a whole group and again for some of the same patients, such as those with follow-up, the patients left out must be able to make up the difference. Using the standard split of a group's variance into its parts, the check flags the pair only when no values for those patients reproduce both means and SDs, allowing for the printed rounding, either way of computing an SD, and up to 15% of the subset's values missing unstated. It runs only on standard deviations, and only where one sentence prints both groups' sizes in a set form that says the smaller group is drawn from the larger one: “104 of the 120 patients”, “Of the 120 patients, 104 completed follow-up”, “120 patients were enrolled, of whom 104”, “120 patients were enrolled; of these, 104 completed follow-up” or “104/120 (86.7%)”. Any other wording is skipped. Only the sentence's wording is checked: the check does not confirm that the smaller count belongs to the larger group, or which group each pair of figures describes. It never runs on a mean printed with a minus sign or on a quantity named as a change, a difference or a delta. | a mean ± SD for a group and for a stated subset of it | Moderate |
| Subgroup difference vs the interaction test | A paper reports one subgroup significant and another not, and treats that as evidence the effect differs between them. Recomputes the interaction test from the two subgroup estimates and their intervals — a difference in significance is not a significance of difference. | two subgroup estimates + their CIs | Moderate |
| Non-inferiority margin vs the interval | A stated non-inferiority conclusion must clear the paper's own margin: the relevant confidence bound has to stay on the non-inferior side of it. Recomputed against the margin as printed, and only when the margin's own position agrees with the direction the paper states. | a margin + the estimate's CI | Moderate |
| “No difference” vs the stated important difference | A “no difference” conclusion whose interval still reaches the minimal important difference the paper itself states for that outcome — the data remain compatible with an effect the authors called important. Only runs when the paper prints that threshold. | a CI + a stated MCID | Moderate |
| Events per variable | Recomputes events per predictor for a logistic or Cox model and reports it against the conventional ten — a long-standing rule of thumb, not a threshold of validity. | events + predictors entered | up to Moderate |
| Fragility index vs loss to follow-up | For a significant binary outcome, counts how many patients would have to change outcome to make the result non-significant, and compares that with how many the study lost to follow-up. | a 2×2 table + patients lost | Moderate |
| Participant flow arithmetic | The CONSORT flow has to add up: those excluded plus those randomized against those assessed, the arms against the randomized total, and each arm's received and analyzed counts against its allocation. Counts that add up to more than their total are moderate. Counts that fall short are mild where they should account for everyone at that step (excluded plus randomized, received plus not received), and are otherwise reported as missing data, since patients lost to follow-up or an arm left out of the flow can leave a gap. | the participant flow counts | up to Moderate |
| Heterogeneity (I²) from Q | Recomputes I² from the printed Cochran's Q and its printed degrees of freedom, and flags a reported I² the pair does not support. Both numbers must appear in the paper. The number of studies pooled is not a substitute for the degrees of freedom, because whether that substitution is valid depends on which Q was reported. | Reported Q, degrees of freedom, and I² | Moderate |
| Sample size vs the stated design | Recomputes the required sample size for a two-arm parallel comparison from the paper's own stated effect size and variability, at the alpha and power it states or the field conventionally assumes, and flags a planned size below it. Always judged against the one-sided requirement, the smaller of the two, so no study adequate under either reading is flagged. Uses the same closed forms as the study planner; paired, crossover, cluster and multi-arm designs are out of scope. | the sample-size paragraph | Moderate |
“Surgical-site infection occurred in 18 of 90 patients (20.0%) in the bundle group versus 27 of 90 (30.0%) with standard care (χ² = 4.83; p = 0.028).”a constructed specimen — and this sentence is the check’s entire input
| study arm | SSI | no SSI | total |
|---|---|---|---|
| bundle | 18 | 72 | 90 |
| standard care | 27 | 63 | 90 |
| total | 45 | 135 | 180 |
A worked example of an impossibility check: the GRIM test for impossible means — how a reported mean of an integer-scored variable can be arithmetically impossible for the stated sample size, no raw data required.
§03 Reference & submission-integrity checks
What is cited, and what was left in the draft. Only the reference identifiers leave the worker — never your manuscript text.
| Check | What it catches | What it needs | Max severity |
|---|---|---|---|
| Reference resolution | Cited DOIs and PMIDs are resolved against Crossref, doi.org, and NCBI PubMed; identifiers that do not resolve — or resolve to a different article — are flagged, with the date checked. Identifiers past the per-manuscript cap, or that a registry does not answer for in time, are listed as not checked rather than passed. | the reference list | up to Serious |
| Retracted references | Each cited DOI or PMID the reference layer reaches (it caps a manuscript at 150 identifiers) with a PubMed or Crossref record is checked for a retraction on that record — PubMed's own indexing and Crossref's Retraction Watch relations. A retracted citation is flagged, with the notice and date when the registry provides them; an expression of concern is reported, not graded. | — | Serious |
| Leftover AI-assistant text | Scaffolding a chatbot leaves behind — “as an AI language model”, “regenerate response”, and the like. | — | up to Moderate |
| Placeholder text | Draft placeholders left in the submission — “[insert …]”, lorem ipsum, an uppercase TODO. | — | up to Moderate |
§04 Reporting-consistency screens
Lighter hygiene — presentation issues a careful reviewer would also raise. They do not change your result; they signal how carefully the numbers were reported. We keep them separate on purpose, so they never dress up as forensics.
| Screen | What it flags | Severity |
|---|---|---|
| Impossible p-values | A p-value printed as an exact zero (p = 0.000); a p-value cannot be exactly 0 — report p < 0.001. | Mild |
| Threshold-only p-values | Results that only ever report a threshold (“p < 0.05”) and never an exact p-value. | Mild |
| Over-precise p-values | A p-value carried to four or more decimal places. | Mild |
| Percentages with no denominator | A percentage reported with no n / N stated anywhere. | Mild |
| Over-precise percentages | Two-decimal percentages from a denominator under 100 — more precision than the data carry. | Mild |
| Unlabeled ± spread | A “mean ± value” with no label saying whether the ± is an SD, an SEM, or a CI — they differ by ×√n. | Mild |
§05 Beyond arithmetic: the methodological appraisal
The deterministic checks above catch what is arithmetically wrong. Judging whether the design supports the claim takes reading — so two independent engines (Anthropic’s Claude and OpenAI’s models) appraise the manuscript blind to each other, across six domains, and a separate literature layer positions the manuscript against retrieved prior work as a seventh. The appraisal is repeated across several passes, and findings generally need to recur — a one-off observation does not become a verdict, while a deterministic discrepancy is reported directly. The two reads are reconciled into one consensus, and disagreement is surfaced, not hidden.
A serious or critical cross-check-only finding must then survive adversarial verification — the engine is made to argue against its own finding, and anything it cannot defend is withdrawn or right-sized before the report reaches you. The deterministic results are merged into that consensus verbatim: a recomputed number can raise a grade, never lower one, and the adversarial pass cannot talk a calculation away. The full method — the seven domains and the claim–evidence calibration headline — is on the methods page.
A seventh domain — contribution & literature positioning — is populated separately, not by the two blind engines. A retrieval layer pulls related prior work, and a literature-grounded appraisal weighs your novelty and framing against it: overstated firsts, and priors the manuscript does not acknowledge, are flagged and graded like any other finding, each tied to the retrieved evidence.
§06 What these checks do not do
Recomputation works only where the underlying numbers are in the text — a reconstructable 2×2 table, integer-summed means, a stated denominator, a labeled spread. RigorMD does not re-run your regressions, hazard ratios, survival models, or other multivariable analyses from the manuscript alone; those need your raw data. It does not detect fabrication or fraud, and it is not a substitute for a qualified biostatistician or for peer review.
Our worked review of the corrected PREDIMED paper shows how a disclosed design limitation informs the assessment of the conclusions. It is one case, not an estimate of detection performance.