[This is the fourth piece in a miniseries on finding the right line between human thought and AI assistance, focusing on the stages of scholarly work from initial ideas through the research process to publication, although I believe much of this discussion is applicable to intellectual work beyond the academy. The miniseries began with this introduction and was followed by an essay on the origins of new ideas and a piece on analyzing evidence and data in the early stages of research. In this issue, I look at handing the entire writing process over to AI.]

At the end of the last essay in this series, on the application of AI to analytical sections of a research project, such as data explorations and visualizations, I left this question dangling:

Why not go further or even all the way? Why not have AI do the entire analytical process and spit out the result, perhaps as a nicely formatted paper?

Point your favorite LLM at a stack of articles, documents, lab notebooks, or data sets, and give it a prompt — or if you really want to go all in, have it come up with its own question to answer or theme to pursue based on an initial pass — and sit back in your comfy armchair as the words flitter by.

This new form of armchair scholarship, like the nineteenth-century academics who wrote books and articles based only on what was readily at hand (in, say, their posh home libraries), rather than doing iterative and extensive field work, lab work, or any other kind of time-consuming engagement with their subject matter, is far from hypothetical. As you read this, AI is probably writing hundreds of academic papers. This time the armchair has a jet engine on the back.

A nontrivial and growing percentage of submissions to journals are now partially or mostly the product of AI, especially in scientific fields. In one study, based on STEM articles from 2021 through 2024, AI usage in the writing of academic articles spiked beginning with the release of ChatGPT in the fall of 2022, quickly reaching over 20% in computer science and electrical engineering, and, to a lesser but still significant extent, in other fields by the fall of 2024. These measurements are surely much higher two years later, especially with the latest agentic AI tools and more advanced models.

These early-adopting AI-assisted scholars are still likely in the minority, and if we want to be generous to them, there may be understandable reasons for their reliance on AI, such as the predominance of English in science publishing. (According to the study, researchers from countries where English is not a first or second language use AI to compose articles at significantly higher rates.) Of course, there’s also the primal need to publish or perish in academia, which has always incentivized cutting corners.

Whether AI authorship of papers is an activity dominated by mercenary researchers seeking tenure or the byproduct of AI translation, the temptation to use AI in the production of scholarly writing will undoubtedly continue to grow. This year has seen a proliferation of websites and software geared toward rapid paper generation. Generally the producers of these tools frame the process as a productive collaboration between you and the AI, but the amount of “you” seems to shrink as you look more carefully and notice the many opportunities to opt out of deep intellectual work. For instance, the AI paper-writing assistant Gatsbi nods toward the discrete stages of research and writing I’ve covered in this series, but also notes that it can go ahead and “auto-draft” the entire paper, stem to stern, if you’d prefer to get a coffee. CoPaper.AI, from Stanford, similarly emphasizes that the scholar guides the production of the article at each turn, but also claims in a large font on its home page that it can take only 20 minutes to sprint from raw data to a final paper. It may be “human in the loop,” but at that scale you’re an ant inside a hula hoop. And naturally you can use Claude or ChatGPT to do end-to-end research and writing, at various levels of engagement from micromanager to laissez-faire napper, whether you’re a fifth grader or a faculty member.

* * *

Call me an optimist, but I believe that the majority of scholars, despite being under enormous pressure to publish, would prefer to be more engaged than removed from the fundamental pursuits and texture of their discipline. They enjoy wrestling with sources, data, and theories, and are innately repelled by the superficiality of having AI write a complete paper. They want to lean into their work, not lean back in an automated armchair. As NYU astrophysicist David Hogg recently wrote, “Anyone working in astrophysics is someone who wants to do astrophysics, not someone who wants to learn the answers.”

Anthropic and OpenAI seem to understand this now, after years of touting AI in a less-than-reassuring way as a replacement for human thought, endeavor, and employment. OpenAI’s new writing tool Prism, centered on LaTeX, a nerdy markup and formatting standard for STEM articles, and Anthropic’s new workbench Claude Science, seem inclined toward more exploratory, iterative, and assistive modes of STEM article generation. Partners, not proxies.

But how slippery is the slope? If one uses AI to help out with part of a scholarly work — for instance, the often (but not always) formulaic “methods” section of a scientific article — will the temptation rise to use it for other parts, such as the more intellectually stimulating and important “discussion” section, in which the results of an experiment are unpacked and its implications for the field made clear? And if so…is that so bad?

Economist and AI enthusiast Tyler Cowen and others working in the more data-centric areas of the social sciences and natural sciences have pondered this question and begun to sour on the old method of article production, instead believing that the future of scholarship may indeed lie in a data set — perhaps one that is constantly updated — and an AI front end that interprets this data. The careful wordsmithing of a paper might be secondary to this direct computational approach, or vanish altogether. And maybe the AI can create the data set too, leaving more time for coffee breaks.

My nagging worry is this: based on a passing familiarity with human nature, I can foresee that an increasing number of academics are going to have to be lashed to the library stacks to resist the AI sirens, who will sing not about more measured uses of AI, but about that sweet, comfortable armchair in which to rest while entire articles are tirelessly generated for them.

* * *

Since we do not have beeswax to put in our ears to resist this song, it seems helpful to examine exactly why the AI generation of a complete scholarly work, rather than using AI judiciously for certain scholarly tasks as I have been arguing in this series, is a bad idea — not just for academic disciplines but for the academics themselves.

At this point, you might be expecting a long rant on AI hallucinations and the possibility of scholarship turning from the pursuit of truth into the extrusion of plausible-sounding truthiness. Hallucinations do remain an area of concern, but it is a problem that has waned over the last year. As LLMs have become more agentic than static, and more rigorously structured in their processes — not relying as much on their initial training set, and venturing out to read external sources of information as needed, instead of immediately starting to spit out text following a query — the number of glaring, or even small, errors has decreased. Especially in the highly connected academic AI environments I have been discussing in this series, with actual libraries available to the LLMs — Claude Science, for instance, can retrieve peer-reviewed research and vetted data from dozens of highly specialized academic resources — the hallucination problem has receded further.

At the same time, hallucinations have not totally disappeared, and academic research should always aim for the highest level of reliability, which makes even the small possibility of hallucinations a shadow over the scholarly enterprise, not to mention the embarrassment of AI-generated faux pas to scholars who take an automated shortcut. Of course, human intelligence can also make errors, or worse, engage in statistical shiftiness or outright fraud. (See the replication crisis.) But we should be aiming to level up on veracity, not down.

More problematic to me than the specter of hallucinations, however, are the long-term effects of the armchair overuse of AI on three areas dear to the academy: the nature of writing and reading, the composition of the sources that writing is based upon, and the enervation of the scholarly mind and scholarly disciplines.

* * *

In early 2023, soon after the release of ChatGPT, I asked, “Can Engineered Writing Ever Be Great?” My answer came from a simple point that every writer knows:

Good writing isn't just the selection and ordering of words, the output; good writing is the product of good reading. Writers aren't indiscriminate generalists, but tend to be rather choosy and personal about what they read. As humans they also have a fairly limited reading capacity, which means that their styles are highly influenced by idiosyncratic reading histories, by their whim.

LLMs, on the other hand, are insatiable omnivores, ingesting as much text, indiscriminately, as they can. This allows them to create countless styles of writing virtually instantly, that feed every need from an automated email response to poetry. But this also means that off-the-rack writing from an LLM tends toward the anodyne, or “slop” if we want to use a more disparaging term.

Over three years later, however, LLM output can be significantly improved, especially if you tailor the inputs to the underlying model, or add post-training context. In one recent study, readers preferred the writing from an LLM trained on books (rather than text that largely comes from the web) over that of human writers with MFAs. I have used Claude Code to create a database of all of my writing (books, academic and popular-press articles, blog posts, this newsletter, and unpublished writing, 2+ million words), which I mostly use to remember and find things I’ve written, but which Claude could also use to compose new pieces very much in my voice, if I wanted it to. (I don’t. The em dashes you often see in my writing are my own; I do love them and don’t care if they have become a marker of AI writing.) If you think that my writing rises above generic AI slop, then I can assure you it’s now possible to use AI to create prose that mimics this more personal, angular style.

Nevertheless, having AI generate an article, even in one’s own voice, inevitably cedes critical intellectual ground. Implicit in the new AI paper-generation tools is the assumption that specific word choices in an article or book are of lesser value than the overall interpretation of sources or data. That may be true in a general sense, and surely many readers of academic articles, ahem, skim, but picking words carefully can increase an article’s power of persuasion and impact. As Daniel Kahneman has shown, a lamentable aspect of human psychology is that we have trouble accepting data as proving a point; we often need well-crafted words and a coherent narrative mapping cause and effect, preferably from someone we see as a peer, to convey the significance of that data and incline readers to accept conclusions. (Even then, alas, human beings can be truly stubborn in their views.) I can now have Claude produce prose that sounds like me, but only the real me can pick the exact words with the right spin and force I’m looking for in a particular sentence. (Plus, I actually enjoy writing; you’ll have to pry my keyboard from my cold, dead hands.)

Furthermore, if we know that a significant percentage of articles are machine-written, we are going to move from careful reading to frequent skimming to a complete abstention from the scholarship in our field. (We will probably have an LLM summarize it for us.) This will obviously greatly harm the exchange of ideas. The only way out of this conundrum is for most practitioners in a discipline to commit to putting in the time and energy to produce and digest thoughtful work. Knowledge production is inherently social — not in the postmodern, constructed-out-of-thin-air way, but embedded in a communal process in which we come to respect, or at least recognize, that other intellects are wrestling with the same problems we care about, and which fosters a continued interest in our common research.

* * *

Then there is the invisible loss of context and detail when AI writes the majority of an academic work. An experiment as a case in point: One day I tried, like an Oliver Sacks case study, to be my wife, who is a scholar of early childhood programs. Using Cowork, I told Claude I wanted to write a paper on how different state policies on child care impact American children and their families. We (Claude and I) decided to use Policy Commons’ invaluable data set, through which Claude assembled a list of 232 state statutes and regulations. Sipping my coffee, I asked Claude to read and process all of these lengthy documents and create a matrix for me so I could quickly assess the contours of child care programs in the United States. But Claude was more caffeinated than I was: like a teacher’s pet it went off and produced, independently, a comprehensive report, not just a table, in a few minutes. It even created its own categories of differentiating metrics, such as infant/toddler:adult ratios, total care group sizes, licensing requirements, academic qualifications for practitioners and directors of programs, sleep protocols, and idiosyncratic state mandates. A few more sips of joe, and a few more prompts to acquire additional data, and I was swiftly on my way to what I thought was a decent meta-review essay.

Now a giddy AI-assisted dilettante, I showed this effortless production to my wife, who proceeded, as an actual expert, to dissect it ruthlessly. Claude got the empirical data mostly correct — no silly hallucinations — but its attempts to extrapolate from the numbers into trends and impacts were clumsy and overly broad. Since my wife actually knows these early childhood programs well, she understands how, on the ground, actual child care sites might differ from written state policies and other sources of information, and she could identify this missing context and additional key details that were invisible to me and my AI buddy. Our armchair scholarship was no match for her decades of experience and knowledge.

The output seemed so good, though…I could totally imagine a less scrupulous academic trying to publish it. The ease of generating an AI paper this way means we will increasingly end up with papers written on the data that is readily available, without questioning how good the data actually is. We will think less about what’s missing.

Yet even with agentic AI — again, Claude Science can reach out to dozens of research databases for material to work with — there are yawning gaps. At the closing plenary this spring at the Coalition for Networked Information meeting in Salt Lake City, “Harnessing the Data Renaissance for Scientific Discovery,” Manish Parashar, Executive Director of the Scientific Computing and Imaging Institute and Chief AI Officer at the University of Utah, highlighted the major work that still needs to be done to create truly rich and comprehensive data sets for many fields, such as cancer research. If we ignore this complex and time-consuming process, and just use AI on top of the data that’s close at hand, we might produce scientific articles more quickly, but we may not make significant breakthroughs. (This is probably already happening.) Parashar, instead, is working with other scientists on a National Data Platform that will aggregate and normalize thousands of sources, without which any AI processes will suffer from a problematic narrowness. In other words, being a good librarian — someone who finds, catalogs, assesses, and merges sources into a coherent and usable library — has become even more valuable.

* * *

Finally, using AI to generate new scholarly papers assumes an iffy theory of intellectual history, that new ideas and discoveries are always implicit but not yet articulated in the existing literature and data. What if new ideas instead come from unique circumstances, lived experience, group interactions, or an individual’s eccentric way of reading and seeing? Yes, the history of science has a number of examples of innovations that seem to be “in the air” and thus “discovered” by multiple people at roughly the same time, such as calculus (simultaneously developed by Leibniz and Newton). But there are many more examples like the one I wrote about in “Can AI Prompt Us to Ask New Questions?,” where a quirky combination of a person’s biography and interests, embedded in a particular social scene, leads to revelations like fractal geometry.

Intellectual history contains both kinds of discoveries, but we wouldn’t want to block the latter, deep river of innovation, and if the overuse of AI in the production of scholarship reduces the velocity of the human mind and the vitality of intellectual scenes, we might find this source slowly drying up. It is worth remembering that asking good questions is harder than giving great answers. Insightful out-of-the-box approaches, often stemming from unusual, previously unasked queries, are the dark matter of human thought and progress, and they often come from the random interactions and odd interests of particular human beings. It is unclear how AI will replicate these uncommon vibrations and collisions.

In “Illegible Benefits,” a piece by Carlo Cordasco of the University of Manchester that is largely positive about using AI in the production of scholarship, he mentions a lingering concern:

I want to be honest about the costs. My ability to hold together a complex position verbally, under pressure, in a seminar or a conversation, has probably not improved and may have declined somewhat. When preliminary exploration is cheap, you spend less time grinding through arguments from first principles, a grinding that builds fluency that shows up in live exchange.

This is the professorial equivalent of the cognitive decline that we worry about with our students who have been using AI for years — and what atrophies in the seminar room will surely atrophy on the page as well. The solution seems clear: if you would like, use AI for the parts of scholarly work where it excels and check all automated output, while retaining human seniority in orchestrating and expressing the meaning of your research.

It’s going to be tough, though. Armchairs really are comfortable.