Science communication · A 2024 study

AI-simplified science: readers learn more, trust is mixed

A scientist's plain-English summary uses only 1.98 points more everyday words than the abstract. GPT-4 beat the scientists' own summaries by 5.69 points. Then two experiments, with 274 and 250 readers, tested what people made of it.

What readers learned and what they thought of the scientist did not move together.

vizmaya desk · 5 November 2026 · Based on David M. Markowitz, PNAS Nexus, 2024

Library of Congress Main Reading Room. Photo: Carol M. Highsmith / Wikimedia Commons, Public domain.

1 · The gapDocumented

A scientist's plain-English summary is only 1.98 points plainer than the abstract.

Since 2012, the journal PNAS has asked authors for a second, plain summary. It is called a significance statement. Researchers checked 34,584 papers that have both.

Everyday words made up 69.77% of the plain summary and 67.79% of the abstract. That gap is 1.98 points. The author calls such gaps small.

How "everyday" is measured: the share of words found in a standard list of common English words (p3). It is a stand-in for plain writing, not a full test.

Could a machine close that gap?

2 · The machine testDocumented

GPT-4 used more everyday words: 75.53% vs 69.84%.

The team gave GPT-4 the same instructions PNAS gives its authors. It wrote a plain summary for 800 papers, and the scientists' own 800 summaries were the comparison.

Scientists beat their own abstracts by an effect of 0.31. GPT-4 beat the scientists by 0.58, which is 1.87 times bigger (computed).

Contested wording: the abstract says the effect more than doubled. On the main score it is 1.87 times (computed). Writing style did not change: 92.73 vs 92.32, p = .246 (p4).

But would real readers notice?

Two hills, sliding apart

Curves are sketched from the reported means and spreads (computed). The real data are lumpier (Fig 1, p17). Source: p4.

PNAS: mean 69.84, SD 7.45. GPT-4: mean 75.53, SD 5.57. 800 summaries each.

3 · The testDocumented

Next, 250 readers each read 5 summaries, with the version picked by chance.

A first test had 274 readers. The bigger one had 250 and added a quiz question and a write-it-yourself summary.

Each reader saw 5 papers out of 20. For each paper, chance picked the GPT-4 version or the original PNAS version.

Readers were US adults recruited on Prolific, average age about 37 (p7). The quiz question was written by one AI model and checked by two others. All three gave the same answer (p7).

So what did readers think of the scientist?

Each square is one reader

US adults250 readers5 of 20 pairs each

Read one summary, picked by chance: GPT-4 or PNAS.
Rate the scientist who wrote it.
Answer one quiz question.
Explain the paper in your own words.

Source: p7. First test: 274 readers, 3 of 5 pairs each (p5).

4 · The scientistContested

Readers judged the scientist two ways at once.

The plain version made the writer look less smart. Intelligence ratings were 5.07 vs 5.24 out of 7 in the bigger test, and 5.00 vs 5.15 in the first.

On trust it split. In the first test, plain text won on credibility (4.72 vs 4.47) and trust (4.70 vs 4.42). In the second, those gaps shrank to 4.81 vs 4.73 and 4.71 vs 4.64, and they were not significant.

Contested: the headline says AI improves how scientists are seen. The second test did not repeat the trust gain. The author guesses the 20-pair design watered it down (p9), but that is an idea, not a result.

Did readers know which text came from a machine?

Ratings of the scientist, out of 7

First test, 274 readers

Second test, 250 readers

Filled dot: GPT-4 plain text. Ring: PNAS original. Source: Table 1, p16.

The axis is zoomed to 4.2 to 5.4. The real scale runs 1 to 7.

5 · The guessDocumented

Readers thought the hard text was the machine's.

The plain text really was written by GPT-4. Yet readers rated it more human: 4.80 vs 4.29 in the first test, and 4.58 vs 4.27 in the second.

They rated the original PNAS text as more likely to be AI: 4.10 vs 3.68, then 4.08 vs 3.90.

Readers did not know who wrote each summary, the author says in a later explainer. The paper reads the result as complexity looking more AI-like than human (p6).

Did the plain text help people learn?

6 · The resultDocumented

Readers explained the science better after the plain version: 0.92 vs 0.72.

After reading, people wrote their own summary. Two AI graders scored it, and plain-text readers got 0.92 vs 0.72. The combined score was 0.19 vs -0.22, and these readers also wrote with more detail and more concrete words.

The quiz question alone did not split the groups: 0.61 vs 0.40, p = .112. The win comes from the explain-it-yourself task.

The two AI graders agreed at 0.70 (p7). Readers were tested right after reading. The paper describes no later test.

So what should anyone do with this?

Score after reading, second test (250 readers)

Explain it yourself: GPT-4 text0.92

Explain it yourself: PNAS text0.72

Quiz question: GPT-4 text0.61

Quiz question: PNAS text0.40

Bars start at zero. The paper does not state the score scales. Source: Table 1, p16.

The quiz gap was not significant (p = .112).

7 · What to doInferred

Hard words looked smarter (5.24 vs 5.07). Readers explained less after them (0.72 vs 0.92).

If you write about science, pick the everyday word and the short sentence. Then ask a stranger to explain it back in their own words.

If you read an AI summary, open the original too. The author warns that readers may read only the summary (p10).

Inferred: these tips are our reading of the paper. It did not test them. Nobody tested a human writing in the same plain style.

Where could this be wrong?

Three moves

Pick the everyday word. The paper counted common words as a sign of plain writing (p3).
Cut the sentence. Word length and sentence length also fed the score (p3).
Ask a stranger to explain it back. That was the test readers took (p7).

Sources: p3, p7, p10.

8 · The limitsContested

Five reasons to hold this loosely.

The plain text was always GPT-4 and the hard text was always the scientists', so wording and writer travel together (p5, p6). The test pairs were the ones with the biggest wording gap: the top 5, then the top 20 (p5, p7).

Machines graded the explanations (p7). One journal and one AI maker were tested (p10). We found no repeat of the study in two searches.

The author warns that simple writing may be read as less important, and that models can carry bias (p10). Missing: any test where humans write both versions.

A clean test would use human writers for both versions. Has anyone run it?

Numbers

Four numbers to keep.

34,584

PNAS papers with both a plain summary and an abstract.

75.53 vs 69.84

Percent everyday words: GPT-4 vs the scientists' plain summaries.

0.92 vs 0.72

Explain-it-yourself score: plain text vs original text.

Table 1, p16

5.07 vs 5.24

Intelligence rating out of 7: plain text vs original text.

Table 1, p16

Every number, and where it comes from

First value is the GPT-4 or plain text. Second is the PNAS original, unless a row says otherwise.

What	Value	Where
PNAS papers pulled vs kept	42,022 vs 34,584	p2
Everyday words, plain summary vs abstract	69.77% vs 67.79%	p3
Gap (computed)	1.98 points	from p3
Everyday words, GPT-4 vs PNAS plain summary	75.53% vs 69.84%	p4
Gap (computed)	5.69 points	from p4
Effect on simplicity score, GPT-4 vs scientists, and scientists vs abstract	0.58 vs 0.31	p4, p3
Ratio (computed)	1.87 times	0.58 / 0.31
Writing style, GPT-4 vs PNAS	92.73 vs 92.32 (p = .246)	p4
Readers, first and second test	274 and 250	p5, p7
Intelligence, GPT-4 vs PNAS	5.00 vs 5.15, then 5.07 vs 5.24	Table 1
Credibility	4.72 vs 4.47, then 4.81 vs 4.73	Table 1
Trustworthiness	4.70 vs 4.42, then 4.71 vs 4.64	Table 1
Seen as human	4.80 vs 4.29, then 4.58 vs 4.27	Table 1
Seen as AI	3.68 vs 4.10, then 3.90 vs 4.08	Table 1
Comprehension index (second test)	0.19 vs -0.22 (t = 5.80)	Table 1
Explain-it-yourself score	0.92 vs 0.72 (t = 8.34)	Table 1
Quiz question	0.61 vs 0.40 (p = .112)	Table 1
Citing works, 2024, 2025, 2026	3, 17, 20	OpenAlex

Method and caveats

Fact: numbers come straight from the paper's text and Table 1. We matched the headline values to the published PNAS Nexus version, and they agree.

Computed: the 1.98, the 5.69 and the 1.87 times are our arithmetic from numbers in the paper. Averaging the three sub-scores instead gives 2.11 times. The hill curves in scene 2 are sketched from reported means and spreads.

Interpretation: the tips in scene 7 are our reading. Nobody tested them.

Left out: we did not read the online supplement, the quiz texts or the OSF data. We left out reading-ease scores, because we could not check how to read them from the paper alone.

Who wrote what: GPT-4 wrote the plain text and scientists wrote the PNAS text. The author's later explainers say readers did not know which was which.

Sources and credits

Markowitz, D. M. (2024). From complexity to clarity: How AI enhances perceptions of scientists and the public's understanding of science. PNAS Nexus, 3(9), pgae387. doi.org/10.1093/pnasnexus/pgae387
Free copies: PubMed Central and arXiv preprint.
Preregistrations: first test and second test. Data: OSF.
Author's explainers: News-Medical, 22 Nov 2024 and ASBMB Today, 12 Jan 2025.
Citation counts: OpenAlex record, read 5 Nov 2026.
Photos, all from Wikimedia Commons: Library of Congress Main Reading Room (Carol M. Highsmith, public domain); Philosophical Transactions title page (public domain); Mechanical Turk engraving (Nhthanhtruc103, CC0); Magnifying glass (Heptagon, public domain).
More from vizmaya: browse the story library.

vizmaya desk · 5 November 2026. If a number here does not match the paper, the paper wins.