Tokyo, 2013. Corporate has flown in the Wall Street Journal’s first social media editor for Asia to get our bureau on Twitter. As the youngest full-time reporter, I rarely get to feel smug. Today I do: I’ve been on Twitter for two years. Some of my colleagues can barely type.
As we swap our egg avatars (RIP!) for headshots and agonize over 160-character bios, we think we’re growing our readership. Really, we’re making a deal with the devil. Over the next decade, publishers like the WSJ handed their audiences to Google and social media. The platforms decided who saw our journalism, kept the ad money, and could turn off traffic with a tweak of the algorithm.
Now AI is offering a similar bargain. Will publishers be smarter this time?
This week I spoke to David Buttle, former director of public affairs and platform strategy at the Financial Times and now co-founder of the SPUR (Standards for Publisher Usage Rights) Coalition, a group of publishers like the BBC and the Guardian setting the rules for how AI companies use their work and securing fair pay when they choose to license it. David believes that publishers can avoid repeating history by doing something they’ve rarely done: work together.
Fair warning: this one is a bit wonkier than usual, but stick with it. It’s a fascinating look at how journalism makes money and should make you feel bad for not clicking on an article when it’s cited by an AI chatbot! Edited for clarity and brevity, as usual.
Where did the idea for SPUR come from? What did you see that convinced you publishers needed to work together?
The move of consumer news consumption from print to digital has been pretty catastrophic for news media. There is no longer an alignment of incentives through the supply chain for news. News media still employ journalists, but they rely on other people to distribute their content and on other people to monetize it.
Two years ago, AI seemed to me — and I maintain this — highly likely to be a new disruptive moment in how people get hold of information, including news. The central belief of SPUR is that if news publishers work together at this moment, when there’s this new distribution paradigm embedding in the market, they’ll place themselves in a better strategic position than they were in during the search and social media eras, where a lot of value has been eroded and has accrued to the distribution platforms.
“News media still employ journalists, but they rely on other people to distribute their content and on other people to monetize it.”
What is SPUR doing exactly?
SPUR is focused on two fairly specific objectives. One is to advance technical measures to help publishers control their IP and prevent the value loss associated with unauthorized and unlicensed use. The second is to shape how the licensing market forms so that publishers can get fairly paid.
Central to publishers’ ability to get fairly paid is understanding what’s happening to their content when it’s being used by an AI system. So we’ve developed a telemetry standard, which is a common set of language that defines usage events.
What does that look like in practice?
When you ask an AI assistant for information that it can’t serve from its training data or the knowledge built as a result of its training process, it will go out and retrieve information from the web. Then it will select a subset of that to include in the context window for grounding, and a smaller subset of that will be cited, with a link displayed to the user. An even smaller subset is going to be clicked on — almost none at all. Some applications also display excerpts.
A common definition of those five events — retrieved, grounded, cited, displayed, and engaged — means publishers can make assessments about where they want their content to be available and what price they want to put on it.
The big-picture issue here is how [publishers] balance monetization off-platform with monetization on-platform. If I’m making my content available for consumption off my platform, it follows that fewer users are going to consume it on my platform.
If I’m launching a small publication, my instinct would be to want to completely block AI companies from scraping us or citing us at all. But I can see how a publication like the Financial Times, with its huge archive and subscriber base, might want to be cited and compensated for that. How are publishers weighing those two paths?
At the start there was this perception — which turned out to be right — that there was going to be a limited round of dealmaking and a finite amount of licensing revenue to be secured from the market. So there was a rush to get a deal. These were also the hottest businesses in the world — there was that dynamic as well.
As time has progressed, rational, data-led assessments probably don’t support dealmaking, unless there’s a very significant revenue component, because it won’t compensate you for the potential lost revenue. The problem is that we have vast imbalances in bargaining power: a highly fragmented publisher market and a deeply concentrated buy-side.
Particularly in relation to Google, but not only Google, even if there’s a little bit of traffic coming back [to your platform], if your content isn’t so distinctive that the buy-side can’t get it anywhere else, game theory will lead most publishers to the conclusion that it makes sense to play.
Say I’m the Guardian, and I decide I don’t want to play — I’m going to say no, to the extent I can. Then that small amount of traffic would go to my direct competitor — the BBC or whoever. You’re confronted with this choice: do I get that small amount of traffic, or does it go to my competitor?
For that general-interest stuff, the game theory makes it really hard for publishers. The more specialist and distinctive you can be, the better.
“The problem is that we have vast imbalances in bargaining power: a highly fragmented publisher market and a deeply concentrated buy-side.”
Let’s also talk about Google’s AI summaries. How is SPUR thinking about that, and how are you?
I’d distinguish between what SPUR is thinking about it and what I think about it.
SPUR is engaging in dialogue with Google, focused on the telemetry standard and on getting data back on content usage within those summaries, so that publishers can make a value assessment about whether they want to participate.
Speaking in my personal capacity, Google is responding to a competitive threat in the form of information retrieval moving to OpenAI. It’s trying to maintain its existing content access model, which is a permissioned one; generally speaking, Google only takes and uses content with the permission of publishers — albeit permission that’s extracted coercively. It’s trying to maintain that model while fending off this competitive threat. It needs to do that by creating services that deliver enough information to the user that they don’t take all their information retrieval behavior elsewhere, but still deliver enough traffic back that publishers still play ball.
That said, most of SPUR’s work on IP control is focused on scraping services and those clandestine technologies that work around technical controls.
On the second goal of helping publishers control their IP, which technical measures look promising?
Firstly, it’s a dynamic arms race between attack and defense. Particular solutions will work briefly and then stop working.
Publishers should not aim to create completely watertight defenses — that’s impossible. They have to raise the cost of unauthorized access, so that we create the right economic incentives to pay publishers. There are some very inexpensive means by which the demand side can get hold of content at the moment. Without giving away the trade secrets, the priority for us is going to be helping publishers close down those loopholes first.
In the US, courts are still deciding whether AI companies’ use of copyrighted work counts as fair use. Where do you think the law is heading?
I think it’s important to distinguish between training and real-time access. Last year's US Copyright Office report made that distinction very clear. It suggested that certain training processes, for certain types of content and certain outputs, are likely to fall within the scope of fair use because they’re sufficiently transformative. But it was also pretty clear that summarization of content in real time to create a directly substitutional output is pretty unlikely to fall within the scope of fair use.
I’m not an IP lawyer, but to me as a layperson, with my understanding of fair use, it doesn’t feel like scraping someone’s content and presenting one of their potential readers with a summary of it, without any money going back to the journalistic organization, sits within the scope of fair use.
I’m sure you saw some of the quotes that came out in discovery for the New York Times–OpenAI case. They were pretty startling, and they also suggested that certain actors found it unlikely that this would fall under fair use.
What about the long tail of creators and publishers? Many of my journalist friends are launching small publications or Substack newsletters. How can smaller players protect their IP?
The work SPUR is doing is deliberately calibrated to help the whole industry, and we have members of all sizes. Admittedly, they’re generally institutions rather than individuals, but the standards we’re building will work equally for individuals. The telemetry standard places the technical burden on the demand side. You don’t have to do anything other than look at the data that’s being sent back to you, so there’s no technical lift from a publisher perspective.
This week we announced the New York Times, Agence France-Presse (AFP), The Economist, and a handful of others. As we continue to grow and speak with a single voice, that will benefit the whole ecosystem.
Things are changing almost daily. What will be the biggest challenges for publishers over the next 18 months?
There are different ways the market might play out over the next 18 months. Generally, I don’t think we’re likely to see a precedent-setting judicial ruling that really changes the risk calculation. So the existing dynamic is likely to continue: information retrieval moving from general search to AI, Google responding, and publisher referrals declining as a consequence.
Publishers are going to have to respond by creating the kinds of content that deliver resilience. That will look different depending on what kind of publisher you are, and certain parts of the industry are going to get squeezed by this dynamic more than others. Then there are new sources of risk for publishers. One of the things I’m quite interested in is the rise of always-on assistants.
I’ve been using Instinct to help brief me before meetings, and it boils everything down to about two text messages’ worth of information. Does that worry you?
I’ve got a briefing service from Claude, which I dip in and out of. In the setup, one of the options is to go off onto the web every day and scrape a load of stuff. Publishers aren’t getting paid for that, and they probably don’t know about it.
At the moment, almost all the risk for publishers is oriented around intent-led traffic: someone wants to know what’s going on in Ukraine and goes to Google. These assistants threaten direct traffic. Conceivably, I could wake up in the morning, look at my briefing from Claude and think, “You know what, I’m not going to go to the FT.” So it eats into a whole different set of publisher traffic.
Those applications are pretty fringe at the moment. They’re for very tech-forward early adopters. But if they really cut through, that would significantly steepen the substitution risk for publishers. Over an 18-month horizon, we could see that start to get traction.
And what could go right for publishers over the same period?
A precedent-setting judicial ruling on RAG [retrieval-augmented generation, where AI pulls in live web content] would incentivize the creation of a licensing market.
The other thing I’d say is the UK Competition and Markets Authority’s conduct requirements for Google, if they lead to Google providing better data in Search Console. Of those five usage events I spoke about, the only ones Google tells you about are citation and engagement. That’s kind of useless — you want to know how many times your content has been used, how many potentially substitutional outputs were created for you to get a click, or even the opportunity of a click in the form of a citation. That would be another real win, because of Google’s dominance.
Let’s say I’m a small publisher. What’s one thing I can do now so I’m not taken advantage of?
Beyond protecting your IP, the space for media owners to attract monetizable engagement on their own properties is probably getting smaller because of AI. So have a really clear focus on who your audience is and how you’re serving them, in a way that will be resilient to being substituted by an AI summary. That’s probably going to be things like personality, audio and video, a highly differentiated niche, and coming at questions from a particular viewpoint.
“Have a really clear focus on who your audience is and how you’re serving them, in a way that will be resilient to being substituted by an AI summary.”
I joined the Wall Street Journal in 2012, and we were pushed hard to adopt social media. We thought we were building a direct connection with readers. Really, we were giving our audience away. What should publishers do differently this time?
Publishers could be thinking about where their optionality lies in relation to traffic sources — this isn’t a SPUR thing. How could they work together to create networks and ecosystems where users flow between premium publisher properties, and reduce their reliance on the type of platforms where they have to put very expensive content, produced by real journalists on the ground in dangerous places, next to user-generated content without any of the checks?
SPUR is a signal that publishers are willing to work together in a way that, until recently, may not have happened.
If you’re in London on Oct 29th, join Alys Key and me for an Oxford-style debate about whether or not AI is making us less creative. We’re nearly at capacity, but if you email me and say you’re a newsletter subscriber, I can make sure you’re on the list.