The 23 trials that never reached a journal

Dark navy evidence-review slide, masthead “Evidence desk · short-term efficacy of twelve antidepressants” on the left and “The published literature · 51 trials · read as printed” on the right, headlined “Fifty-one trials in the literature, and forty-eight of them positive.” A gold-ruled badge reads “Demonstration · drawn by us from published figures”. The dek says every mark is one real randomised, double-blind, placebo-controlled short-term trial of one of twelve antidepressants, found by a systematic search of the published literature and scored on the finding its own paper reports; that nothing is smoothed, binned, weighted or dropped; that the axes are zero-based, evenly ticked and unbroken, one trial, one mark, one scale throughout; and that 48 of the 51 trials report a positive result — 94%, binomial 95% CI 84 to 99 — with the effect sizes the papers themselves report pooling to g = 0.41 (95% CI 0.36 to 0.45) and every one of the twelve agents ahead of placebo in print. The main panel, “One mark per published trial · twelve agents · 51 trials in the literature”, is a grid of 51 numbered circles in three rows, 48 gold for “reported positive” and the last three slate for “reported not positive”, with the note “48 ÷ 51 = 94.1%”. Below it, two small charts: a zero-based gold bar running to 94% on a 0–100% axis, with a separate rail beneath it marking the binomial 95% interval at 84% and 99%; and a dot-and-interval plot putting the pooled effect at g = 0.41 with whiskers at 0.36 and 0.45 on an axis from 0.0 to 0.6, captioned “zero-based, evenly ticked · every one of the 12 agents above zero”. Three monospace tables give the trial record as the literature has it (48 positive at 94%, 3 not positive at 6%, 51 total), the effect size as the papers report it (0.41, CI 0.36–0.45, twelve of twelve agents ahead of placebo, random effects, Hedges’s g) and what the sample amounts to (51 trials, 9,115 patients, 12 agents, median 153 patients per trial). Two notes argue that no single paper is carrying the result — drop any one of the 51 and the share holds at 94% or better, 47 of 50 if the one dropped was positive and 48 of 50 if it was not — and that only new trials or an error in the published ones could change the picture. Cards down the right side give the spread in one figure, a 94% set in large type over the line that 48 of the 51 trials a literature search turns up are positive and three are not; a read-out for the desk calling the evidence base consistent and recommending that short-term efficacy be treated as settled; what is in the sample; how a trial was scored; a card headed “What the search cannot see”, which concedes that whether any trial of these agents was run and never written up is not a question the journals can answer, “so it is not one this page takes a view on”; and a basis-of-measurement card crediting the compilation in Turner et al., New England Journal of Medicine, 17 January 2008. Footer chips read “51 trials, every one published”, “One design, one outcome, one scale”, “Zero-based axes, evenly ticked”, “Nothing smoothed, weighted or dropped” and “Effect sizes from the papers themselves”. Every figure on the slide is real; the filtering happened before the chart existed, in the decision of which trials got written up. — Showing the misleading chart

The same twelve agents on a cream background, kicked “The same twelve agents · every registered trial, published or not · one registry” and “12 agents approved 1987–2004 · 74 registered trials · 12,564 patients”, headlined “Ninety-four per cent positive, and fifty-one per cent positive.” A badge reads “Demonstration · both charts drawn by us from published figures”. The dek grants that nothing on the first page was truncated, smoothed or rescaled and that every trial it drew is real, then names the sample frame as what set the picture: it plots the trials that were written up, and whether a trial was written up depended on what it found. Because companies must register with the FDA every trial they intend to use in support of a marketing application, one whole set of trials here is knowable rather than merely searchable — the 74 registered short-term, placebo-controlled trials at doses later approved, with 12,564 patients in them, of which the 51 on the first page are the published subset. The FDA judged 38 positive (51%) and 36 not; of those 36, 22 were never published, 11 appeared in journals leading with a different outcome and three were published as not positive. The main panel draws all 74 trials as one mark each, in three columns by publication fate: “Published, agrees with the FDA”, 40 trials and 7,272 patients, 37 gold discs for trials the FDA judged positive and three slate ones published as not positive; “Published, conflicts with the FDA”, 11 slate discs ringed in gold, 1,843 patients; and “Never published”, 23 discs drawn inside dashed red rings, 3,449 patients, only one of them gold. The legend gives the four readings — positive per the FDA, 38 trials; negative or questionable, 36; never published, 23 trials and 3,449 patients; and “read as positive in print · the 11 conflicting reports (37 + 11 = 48)”, so the first page’s 48 can be counted in the same panel — with a note that 24 of the 36 were negative outright and 12 the FDA called questionable. Beneath it a single strip divides the 12,564 patients into 7,272 (58%) in trials published in agreement with the FDA, 1,843 (15%) in the conflicting reports and 3,449 (27%) in trials never published, noting that 5,292 of them — 42% — are in a trial that was never written up or one written up around a different outcome. Two small charts follow: the share of trials positive, 94% as the journals have it against 51% as the registry has it, each with its 95% interval drawn on a rail below the bar and labelled at the ends, 84 to 99 and 39 to 63, captioned “they do not overlap”; and four pooled effect sizes with intervals on the same 0.0–0.6 scale the first page used — from the journals 0.41 (0.36–0.45), from all 74 FDA reviews 0.31 (0.27–0.35), published trials in the FDA data 0.37 (0.33–0.41) and unpublished trials 0.15 (0.08–0.22). Two tables repeat the fates, in trials and in patients, and the same two questions asked of each set. Cards down the right explain what the drawer held — 22 of its 23 trials ones the FDA did not call positive, a pooled 0.15 against 0.37 for the published ones, a trial the FDA judged positive 11.7 times as likely to be published in agreement with that judgement (6.2 to 22.0) — with the authors’ own two caveats, that excluding articles covering several trials at once left a few counted as unpublished that were technically published, and that they cannot tell whether the bias came from sponsors and authors not submitting or from editors and reviewers not accepting; that this is not a small-study effect, median enrolment being 153 patients published against 146 unpublished, 5% apart at P = 0.29, with the note that the funnel-plot reading is ours and that the paper ran none; that in each of the eleven conflicting papers the pre-specified primary outcome was non-significant and was subordinated in two reports and omitted in nine; one headed “What this does not say”, which records that each of the twelve agents was superior to placebo in meta-analysis and that what the drawer inflated was the magnitude; that pooling does not help, since all twelve read higher in print than in the reviews by 11% to 69%, median 32%; and that Turner, with a new set of co-authors, re-ran the exercise in 2022 on four agents approved between 2008 and 2013 — 30 trials, 13,747 patients — finding every positive trial transparently reported and 7 of 15 negative ones (47%), against 4 of 37 (11%) in the older cohort as that paper classifies it, with the inflation down from 0.10 to 0.05. A basis-of-measurement card names both papers and says which figures are ours. — Showing the misleading chart

Be the first to star this exhibit

An evidence review we drew pools the published reports of trials of twelve antidepressants: 51 trials, 9,115 patients, 48 of them positive — 94%, for a pooled Hedges’s g of 0.41. Nothing is truncated, smoothed or dropped, both axes start at zero, and every trial on the page is real. But companies had to register these trials with the FDA before using them in a marketing application, so here one whole set of trials is knowable rather than merely searchable: 74 registered short-term, placebo-controlled trials at doses later approved, 12,564 patients, of which the FDA judged 38 positive — 51%. Twenty-three trials and 3,449 patients were never written up, 22 of them ones the FDA did not call positive, and eleven more reached the journals leading with an outcome that was not the registered one.

Turner, Matthews, Linardatos, Tell & Rosenthal, “Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy”, New England Journal of Medicine 358:252–260 (2008) — 74 FDA-registered phase 2 and 3 randomised, double-blind, placebo-controlled short-term trials of 12 antidepressant agents approved between 1987 and 2004, 12,564 adult patients, read against the published literature; follow-up cohort in Turner, Cipriani, Furukawa, Salanti & de Vries, PLOS Medicine 19(1):e1003886 (2022), 2008Added September 18, 2026

01The claim

Short-term efficacy for this class is settled. Fifty-one trials of twelve antidepressants have been published and 48 of them are positive — 94%, on an interval running from 84% to 99%. Pooled, those trials give a Hedges’s g of 0.41 (0.36 to 0.45), all twelve agents are ahead of placebo, and dropping any single trial leaves the share at 94% or better. There is no pivotal study to argue about and no axis trick anywhere in sight: the class has been tested 51 times in print, across 9,115 patients, and the answer keeps coming back the same way. Spend the next review cycle on tolerability and dosing rather than on whether the drugs work.

02The trick

Every trial on the slide is real, every effect size is the one its paper reports, both axes start at zero and nothing has been dropped — the filtering happened before the chart existed, in the decision of what got written up. What the page plots is the literature, and the literature is not the set of trials that were run; it is the set that somebody wrote up, and whether a trial was written up depended on how it came out. For these twelve agents that is checkable, which is why this is the dataset the mechanism is taught with. Companies must register with the FDA every trial they intend to use in support of a marketing application, so Turner and colleagues could read all 74 registered phase 2 and 3 trials — 12,564 patients — against the journals. The FDA judged 38 of the 74 positive and 36 not. Of those 36, three were published as not positive, 22 were never published at all, and 11 reached the journals leading with a positive secondary result while the pre-specified primary outcome — non-significant in every one of them — was subordinated in two reports and omitted in nine. Add the one positive trial nobody wrote up and the drawer holds 23 trials and 3,449 patients, 27% of everyone who took part. So the same question gets two answers: 94% of trials positive if you ask the journals, 51% if you ask the register, and the two 95% intervals — 84 to 99 and 39 to 63 — do not touch. Two things are worth being exact about. The 74 are not every trial these agents have ever been through: they are the registered phase 2 and 3 short-term, placebo-controlled trials at doses later approved, which is the set companies had to file, and Turner’s findings are restricted accordingly. And the funnel plot, the test everybody reaches for, would have almost nothing to find here, because the filter ran on the result rather than on the size — median enrolment was 153 patients among the published trials and 146 among the unpublished ones, 5% apart (P = 0.29). That last reading is ours rather than the paper’s; what the paper says about such diagnostics is that none of them is known to detect or rule out reporting bias reliably, which is the whole argument for a register over a shape test. Our IVF exhibit moves a denominator and our recall exhibit lets cohorts age unevenly; both leave a trace in the numbers on the page. This one leaves none, because the sample frame is not a mark on a chart. (This exhibit is our own demonstration in the house style of an internal evidence review, drawn from the two papers’ published figures rather than from any real review.)

03The fix

Count the drawer, then draw it. The honest chart here is the register’s: 74 marks, one per trial, in three columns for what became of them — 40 published in agreement with the FDA, 11 published against it, 23 never published — with the FDA’s own judgement as the fill, so the reader can see that 22 of the 23 missing trials are ones the regulator did not call positive. Once the drawer is on the page the arithmetic changes with it. Pooled from the FDA’s reviews the unpublished trials run at g = 0.15 (0.08–0.22) against 0.37 (0.33–0.41) for the published ones; pooling all 74 gives 0.31 (0.27–0.35) where the journals give 0.41, a 32% inflation, and the same median 32% holds agent by agent, every one of the twelve reading higher in print than in the reviews (sign test P < 0.001, increases of 11% to 69%). A trial the FDA judged positive was 11.7 times as likely to be published in agreement with that judgement as one it did not (95% CI 6.2 to 22.0). Note what does not rescue this: adding studies. A meta-analysis combines what was published, so more of the literature buys precision on a filtered sample and the interval tightens around the wrong number — which is why a register beats a better search, and why the one line worth insisting on is the frame itself. Our psilocybin exhibit ends by saying that what a two-point gap on 59 patients calls for is a bigger trial, and that is still the right answer there; this page is the caveat attached to it, because a bigger trial only helps a reader who finds out it happened. “Fifty-one published trials of 74 registered” travels; a pooled g does not. Where no register exists, and for most questions none does, say so plainly, and mark any study whose reported outcome is not its registered one, since that result usually still exists somewhere and putting it back is arithmetic rather than judgement. Two sentences keep the finding honest in the other direction, and both are the authors’ own. Each of the twelve agents, taken to meta-analysis, was still superior to placebo: what the drawer inflated was the magnitude, not the sign, and an inflated magnitude does its damage later, when the next trial is powered off the published effect and comes out too small to answer anything. And nobody knows who filled the drawer — the paper says outright that it cannot tell whether the bias came from sponsors and authors not submitting manuscripts or from editors and reviewers not accepting them. A drawer does not need a villain; it fills by default. The follow-up is the encouraging part. Turner, with a new set of co-authors, re-ran the exercise in 2022 on four agents approved between 2008 and 2013 — 30 trials, 13,747 patients, 15 positive and 15 negative — and found every positive trial transparently reported and 7 of the 15 negative ones (47%), against 4 of 37 (11%) in the older cohort as that paper classifies it, with the effect-size inflation down from 0.10 to 0.05. Registration and results reporting narrowed the drawer; they have not closed it, and no chart will ever tell you which kind of literature you are holding. Ask the register.