Should politicians write policy papers with AI?
The Economist's Archie Hall on the evolving etiquette around AI and writing
One of the best things I’ve read about AI writing recently is this piece by The Economist writer Archie Hall.
He captured something that I struggle with: how to justify my hostility towards AI-generated writing. I can argue that it’s excruciating to read, but one day, machine-generated prose may be indistinguishable from our own. Does that mean it will be okay to use it then?
I reached out to Archie, and he kindly offered to speak to me about how he sees the debate from his position covering economics and politics—including what looks like AI writing creeping into policy papers. I came away thinking that:
Norms and etiquette around AI will keep shifting until we resolve the gaps between people who can recognize AI text and those who can’t and between people using AI and those who aren’t.
AI writing feels ick, especially in high-stakes writing like policy papers about a country’s future, because its use suggests the writer hasn’t fully developed the ideas behind their writing. Why should people follow lukewarm ideas?
Read on for our conversation and, at the end, some artistic work by yours truly.
How did you start thinking about the AI and writing topic?
These AI models often produce text that people find very appealing if you’re not someone who spends a lot of time staring at AI output. But if you spend a lot of time staring at AI output, you see it as slop. I struggle to read it, and I’m a massive booster of using AI in many parts of my job, which is very writing-heavy.
Where were you seeing AI slop seeping into society?
In some of the policy papers that I was reading. The Labour Party claims to be in a battle of ideas, so everyone wants to have ideas, and ideas mean long essays. That creates massive incentives toward AI use, because you can put out a chunky piece of writing that looks like a very material policy proposal, even if it didn’t take you very long to write.
I put some things through Pangram, and it came back saying a large majority was AI-written or assisted.
I guess some people would argue that creating these proposals quickly to help the population understand the policies being proposed is a good thing, and not everyone is a great writer or likes writing.
It’s true that if you’re a writer, it’s very easy to assume that writing is enjoyable for everyone. And for a lot of people, it’s absolute hell — including us writers.
But many of these policy papers are saying, “These are substantial changes the country should follow.” Those ideas are shaped, to a degree we don’t know, but which is possibly quite substantial, by writing. Any writer will know that your ideas only really get stress-tested if you write them. You can get away with sloppier thinking if you dump some rough notes into an LLM. So I feel queasy about someone then trying to get the country to follow them on those ideas.
Politicians have a responsibility to put forward robust policy ideas and to express them in a way that is easy to understand and digestible. They have a responsibility for both content and expression.
Right now, I think there is a sort of implicit contract with a reader. You are quite selfishly demanding their time, and there’s lots of text in this world. Don’t make it any longer than you have to. Make it digestible.
I write about quite dry economics-y topics, and I think a lot about trying to make it funny and interesting. For instance, I did a piece in The Economist a couple of weeks ago on the Paul Krugman versus Luis Garicano and Philippe Aghion dispute over which purchasing-power-adjusted GDP measure to use to compare the US to Europe. The debate gets to the high-stakes question of: “Was Europe stagnating or not?” But at some level, it’s a debate about an incredibly abstruse technical measurement. So I spent a lot of time thinking about how to get an analogy about the World Cup in my article, finding a few snarkier asides to slip in, and so on. All of us writers have an obligation to our readers to make ourselves digestible. And LLMs certainly aren’t doing that.
AI makes it easier to do wrong by your readers and waste their time, because you can, for instance, now produce a long policy paper — let’s say 20,000 words long. You can dump this tome on the discourse, and because norms have not caught up, a lot of people will go, “Wow, this is a really intellectually important intervention.” Five years ago, this might have taken a team weeks to produce; now you can produce it in a matter of minutes or hours.
How else have you seen that asymmetry of norms appear?
Right now there’s a very binary AI-good/AI-bad axis. For instance, some of the responses I received on social media when I was airing some of these grievances about AI writing were: “Isn’t it ironic that you’re outsourcing your own LLM detection to an LLM?”
There are lots of things that AI is better at than humans, including detecting AI because it’s a quintessential machine learning classification task. So it makes sense to me to use AI when it’s better than humans. That’s one case where we’re still groping towards a sense of where the right boundaries are.
I think the other one is that people don’t have a well-calibrated sense of how easy or difficult some things are, which I think means that people who spend a lot of time using AI tools are getting over-credited.
Yes! I made a demo of a little tool and showed it to some non-AI users who were very impressed, when in reality, it was a one-shot thing I made on Fable 5 that didn’t have real functionality.
We’re still in a world where, if you have a great chart to go with your prose, people think, “Wow, this guy really did the work.” People are calibrated to a time when that was all very laborious relative to now. Data is something I try to do very distinctively, and it’s now a lot cheaper — for lack of a better word. So I think that the prestige and value of data journalism will shift in the next couple of years.
But there’s an upside, which is that AI unlocks very cool and interesting use cases. I fed The Economist leader archive since 2000 to LLMs to score the accuracy of our predictions. I could have done this in the past, hired dozens of researchers, had them all manually grade things with a rubric and graded each other, but it would have cost thousands of pounds and could have been a disaster. The fact that I can do this so easily now is a total unlock.
But even if we can create analysis faster, humans can only consume so much analysis. I’ve edited someone who would record himself talking, use a prompt to turn that into an article, and send it to me for review, when it would have been more time-efficient for both of us had he sent me three bullet points to read and review. What happens when the tools are faster than the timescale we humans live on?
I can make these quite forceful moral pronouncements about AI writing in the context of the society we live in and the norms that we live by, which have evolved over centuries. But potentially this is such a shift that the ways in which we digest and produce information might change more fundamentally.
I don’t think the answer to that is, “Great, I’m fine to dump a bunch of AI slop on the world,” but imagine a world where AI prose gets indistinguishable, at least to the human eye, from human prose, and doesn’t have all the unpleasant tells. I think this is highly likely in the next few years. LLMs could be even better at pulling out the important ideas from a chunk of text and synthesizing how it connects to other things. Do we end up in a world where everything we read and consume is not just written by LLMs but mediated through tiered layers of LLMs?
Somehow I don’t think this world is where we’ll fully end up. I work for a news organization that declares proudly that humans do our work. I think there’ll still be people who want to believe a thoughtful person has worked through something.
Do you think human writing will always have some kind of edge?
I feel very confident that AI will eventually get to a stage where the absolute writing clangers, the really awful constructions, disappear. But I would like to think and hope that a really good human writer will have an edge, and that in a world where we are deluged with mediocre text, some people will be more discerning about saying, “I want to read something really good, because there’s all this stuff out there.”
But maybe the edge will not be in the text; it will be in some combination of thinking, or having a clear set of views, knowing non-public information, or a bit of all of the above.
I also feel that there will be norms for individuals and norms for organizations. An organization might want to draw harder lines, similar to how, as you mentioned, The Economist is very firm in standing by human-created content.
We have a public AI policy that says humans produce all our journalism, but we do use AI for certain things. We have a summary button at the top of the articles. We use AI tools a little bit in video translation. If you’re presenting a TikTok-style clip about your article, which we sometimes do, and you slip up verbally, a producer can correct this with AI. Obviously we’re drawing a boundary, and I think it’s a pretty good boundary.
On the other hand, nothing is theoretically stopping me from churning out AI-written articles on my Substack. But that would be very embarrassing, given what I’ve said on the issue.
I feel like in the rest of the economy, it will come down to how this thing works and what level of human involvement is needed beyond prompting. So I’m quite interested in how something like management consulting as an industry shakes out.
Like whether AI will ever win against PowerPoint! At Every, we tried to automate slide making with AI, and it’s still very hard. We made a plugin for a client that costs $62 to run and involves 24 skills strung together. It’s a bit better than one-shotting it, but definitely out of the ability of most organizations.
I think the goal with AI use here is more plainly utilitarian: Are you giving good business strategy advice or not? There are strong market incentives operating in this kind of industry, and perhaps consultants will be willing to tolerate a little bit of slop aesthetic if they can get the core thing they want, which is “How do I do the work for my client at a fraction of the price?” There hasn’t been a big price war in the industry yet, and I’m interested to see how this will end up.
Was there anything on your AI writing graph that surprised you?
It was actually greener than I expected, in the sense of green representing when it’s okay to use AI.
If you’re trying to either persuade the country to follow your policy ideas or simply persuade readers to agree with you, you owe it to them to make sure you’re tight in your thinking yourself, and that requires a more hands-on approach to the writing than LLMs currently offer.
Archie has also proposed a rubric for determining what we should let AI write, based on how mechanical or ideas-led the writing is and what step in the writing process we’re talking about.
I filled the rubric based on my vibes; similar to Archie’s version, there is a lot of green. Today is not the day I will die for artisanally crafted admin emails.
Where I differ from him is in fact-checking, where I still feel nervous about AI. Frontier models have gotten much more accurate (so accurate that my colleagues at Every laughed at me for bringing up the topic of hallucinations recently), but when my neck is on the line, I can never be too sure. I think this paranoia is linked to my journalism/comms background—cue stress dreams about getting someone’s name wrong.
When writing an article or essay, I use AI to create a list of facts that I can then check manually. I have a fact-checking skill in Claude, which is basically:
Extract every verifiable factual claim from this article. Include:
Names, dates, locations, titles, organizations
Statistics, quotes, events, or assertions presented as fact
Any statement that could be independently confirmed or refuted
Format the output as a numbered list of direct quotes or paraphrased claims with their original context (e.g., paragraph or line number). Exclude opinions, analysis, or hypotheticals.
Feel free to steal my skill, and please send me your own AI writing rubrics! Have a great weekend ☀️