Microsoft CEO Satya Nadella says he’s AGI-pilled, at least when it comes to how AI might make startling changes to the structure of the economy and how politics might change because of it. “I am AGI pilled in the sense that at an economy level, what is the political economy’s organization around AGI?” he asked at a roundtable discussion following his Microsoft Ignite keynote at the Chase Center in San Francisco.
But Nadella doesn’t think frontier models like Claude are conscious. “I don’t think so,” he said, adding that he follows Microsoft AI CEO Mustafa Suleyman’s view that companies should not give rights to models and treat them like they might have a moral status, which companies like Anthropic do as a matter of principle. Suleyman has told me he thinks Anthropic’s team has “wireheaded” themselves, and that this approach is dangerous.
Nadella wasn’t quite as firm about it. “It’s a good debate,” he said. “I’m not sort of dogmatic about it. What I’m saying is the most important social technical issue we have to deal with is how do you ensure that an economy can diffuse this technology broadly?”
I asked Nadella how he was thinking about AI after hearing his response to another reporter’s questions about Anthropic releasing more products in its office suite. Nadella talked at length about the potential collapse of the economy and the end of democracy if one AI model were allowed to achieve monopoly status.
“If the Marxist sort of view is that there’s only one firm, then it’s over, right? For all of us.”
“Most people think about this as a competition between tech firms, which of course it will be,” he said. “But remember, the model has no limits. It’ll be a healthcare company. It can be a financial services company. It can be anything if it wants to be. It can take over the government. Thinking of this as some narrow competition between tech firms, of who is going to get to the next trillion dollars of market cap, is one of the strangest, narrowest ways to think about it.”
Nadella went on to say that the very nature of companies stood to be disrupted in the future. “If intelligence has network effects, what is the future of a firm?” he asked. “If the model sees anything and learns it, and essentially all of human capital collapses into token capital, what is the role of a firm and the human capital in a firm? That’s the debate.”
“I think the political economy is going to get more and more interesting in that context. No one — at least in a democracy — can even win an election if this just goes that way, if there’s only capital and there’s no labor. I mean, I don’t know, that’s not an economy.”
“If the Marxist sort of view is that there’s only one firm, then it’s over, right? For all of us. Not just for me. You are also going to somehow be some affiliate of Anthropic.”
Needless to say, this is not how tech CEOs usually talk about the launch of competing spreadsheet software.
Of course, Nadella was also promoting Microsoft’s various AI tools at Ignite, which promise to let enterprise clients pick the most cost-effective models for various tasks, run their own models, and control their own data in different ways. The idea here is that the market will demand many different kinds of models and sovereign data in order to keep control from centralizing, a theme Nadella kept coming back to as we spoke.
“I think that if you believe that the world, the economy needs more than one firm or two firms, then by definition there need to be as many models as firms of some sort,” he said.