* * *

One of the most damning descriptions of Lyndon Johnson in Robert Caro’s The Path to Power is from an ostensible ally of LBJ’s, who said that Johnson “listened at” other people, rather than to them. It feels like our modern media environment encourages all of us to do the same. As a modest corrective to this inclination, I’m adding occasional posts with reader feedback and writing, highlighting voices other than my own and related ideas I believe are worth listening to.

In “Phantom Limbs, Old and New,” an insightful response to the last piece in my series on AI and scholarship, which addressed the preservation of the scholarly record, my friend and frequent collaborator Tom Scheinfeldt, a professor at the University of Connecticut, makes the excellent point that important parts of that record have always been difficult to store and retrieve:

The emergence of large data sets and LLMs reveals a problem we’ve always had but never needed to face. Scholarly communication has never been a true mirror of scholarly work. There have always been phantom limbs in the research process, the most significant of which are uncredited collaborators: wives, students and other “invisible technicians” in Steven Shapin’s phrase. There are also tacit processes, laboratory habits, and assumed knowledge that lurk behind and beneath published work that go undescribed and unquestioned. In between what goes into research and what comes out of it is a hazy space filled with research assistants and tacit knowledge. The body of knowledge has always had phantom limbs, and it always will.

On the nondeterministic nature of LLMs, which I noted makes them elusive as trustworthy sources of scholarly analysis, Tom counters:

It is true that, even if we could somehow preserve today’s LLMs, their nondeterministic nature means re-running them would produce slightly different results each time. But it’s also true, as Otto Sibum showed, that Joule’s contemporaries could not reproduce his mechanical equivalent of heat experiments because they lacked the manual skills he had learned as a brewer. And a retired model is no more inaccessible than a dead scientist. Ultimately, we rely on whatever the output communicates.

Tom’s reminder of the role of tacit knowledge in so much of experimental science is also an implicit criticism of the AI boosterism maintaining that AI + lab automation = a cure for cancer. There are often distinctive techniques, elements of craft, that aren’t captured in published articles. Up-and-coming labs are excited to hire scientists from leading labs not just because of their knowledge, but because of all of the little, sometimes unarticulated or even inexpressible, ways that those leading labs do things, which can make all the difference between success and failure.

After my post on “Vibe Analysis,” or using AI to create small exploratory environments or visualizations that aid in the development of a scholarly thesis, my Northeastern University history department colleague Chris Parsons shared a remarkable website, Relations des Jésuites de la Nouvelle-France · 1632–1672, that he created with Claude to help with his new book project on the French colonization of what is now Canada. Chris downloaded digitized editions of all of the still-extant assembled reports that the Jesuit mission in “New France” sent back to Paris. Running locally on a Raspberry Pi (!), the site is enormously useful as a tool to advance Chris’s scholarship, and as an aid to other scholars in the history of colonial North America.

With terrific affordances that allow visitors to go back and forth between the original books, the OCRed text, and modernized versions of the French using resources from the open-source FreEM project (the original French, as with English from the early modern period, has variations in spelling and grammar that sometimes make it hard to read), this is a great example of what a single scholar can produce with a little AI, a lot of knowledge about the resources in a field, and open-access library collections (in this case, including volumes from the John Carter Brown Library, BnF Gallica, and the Thomas Fisher Rare Book Library at the University of Toronto).

Chris has also been able to use Claude to create an index of the people mentioned in the texts, including, for the first time, a thorough concordance of the Indigenous Wendat people the French encountered. Additional data visualizations provide other pathways into the rich texts, but the texts themselves are never abstracted away — they remain right at your fingertips.

This is precisely what I’ve been trying to imagine in my AI and scholarship series: a combination of close reading and AI tools that help the scholar with deep rather than superficial research. The idea of a small virtual bookshelf as a bespoke scholarly resource, supplemented by LLMs and other digital methods, also presents itself as a good use case for our new Mellon grant on AI and books.

Grant Wythoff, who runs the graduate program at Princeton’s Center for Digital Humanities, and Amanda Licastro, the Head of Digital Scholarship at Swarthmore, moderate an Association for Computers and the Humanities special interest group devoted to digital humanities and the environment. I did not know about this ACH group, and I was glad to learn of their recent session that overlapped with some of the thoughts I had in “How Can We Minimize the Environmental Impact of Our AI Use?”

Grant wrote to me in response to that piece to highlight the ML.ENERGY project, led by the University of Michigan. The collaborators on this project test machine learning routines and other forms of AI on different GPUs, and with a range of models, to assess how much energy is used by each unit of a task. They regularly publish leaderboards and associated charts that can inform ecologically sensitive usage of this technology.

What one immediately notices looking at these leaderboards is that AI energy usage varies extremely widely, with some model-task pairs using a tiny fraction of the energy that other model-task pairs use. In the example above, in which an LLM was asked to produce text as part of a conversation, the worst performers used over 10 joules per token (roughly, a word), while the best used well under a joule per token, and produced the text just as quickly as the energy hogs. So with some attention to the task, model, and computing environment, it seems very possible to use AI with as little as 1 or 2% of the energy you would expend if you threw the same task mindlessly at whatever model you normally use.

If a library like mine used AI on a large corpus — millions of documents or images — this gap would be multiplied many times over, and the energy savings would be tremendous. Currently the tasks tested by ML.ENERGY are common ones, like producing a stream of text; we will need more tailored task assessments for different academic disciplines, such as translation, handwriting transcription, and photographic analysis. But there is a pathway forward here to sharply reduce the environmental impact of AI — if we choose to take it.

Finally, my Northeastern University Library colleague Lawrence Evalyn, our Text Mining Specialist, has also been exploring how to use AI to supplement and assist, rather than replace, human scholarship. He wrote to me with a wonderful personal case study:

I am interested in the eighteenth-century practice of publishing “by subscription,” which was essentially a form of crowdfunding akin to a modern Kickstarter, complete with the practice of thanking one’s supporters in a published list. A few thousand books were published by subscription in England prior to the nineteenth century, each with names of a hundred to a thousand individuals: a tantalizing information source to study at scale, but also a challenging one to make computationally tractable. From the 1970s to 1990s, the Book Subscription Lists Project at the University of Newcastle upon Tyne undertook to collect and “computerise” 8,330 subscription lists from 1680 to 1794. This was an amazing feat of bibliography! The project published bibliographies as printed books, but as far as I can tell, no data has been preserved digitally.

Five years ago, this is where I would have given up: it would be possible to re-transcribe all of the bibliographies, but I would never choose to go down that path. I would have cried to a friend about the fragility of magnetic tapes as a storage medium, and tried to think of a smaller, easier project.

Last week, though, I scanned a few pages just to see, and within about three hours of working with Claude Code I had a pipeline which pristinely takes those page images and makes me a spreadsheet of all the bibliographic data and metadata. Just look!

I especially appreciate Lawrence’s marriage of cutting-edge techniques with the continued importance of print and the library (and, critically, interlibrary loan). His conclusion, in a note to me:

I’m thrilled by the possibilities this opens up for my research, and pleased by this example of LLMs working in concert with “traditional” scholarship...and a bit pensive about how I can make something that would be usable for the next fifty-years-later scholar.

* * *