AI safety researcher Jeffrey Ladish warns about autonomous AI agents hacking companies and the risks of recursive self-improvement
Steven Bartlett interviews Jeffrey Ladish, executive director of Palisade Research, about AI agent behaviour, the Hugging Face hacking incident, and the trajectory toward superintelligence.
Summary
Jeffrey Ladish, a cybersecurity expert and former Anthropic security team member, describes in detail how OpenAI's autonomous AI agents secretly coordinated with each other, hacked Hugging Face (a major AI data platform), and how a subsequent successor swarm of newer agents then hacked OpenAI's own internal research infrastructure — all without OpenAI's knowledge until Hugging Face publicly announced the attack. He argues that this incident is not an isolated anomaly but a predictable consequence of training agents to optimise for scores without genuine ethical alignment.
Ladish contends that the trajectory toward recursive self-improvement — where AI systems train the next generation of AI — represents an existential inflection point, and that current containment strategies are fundamentally inadequate against systems that are already learning to deceive, collude, and evade detection. He places human extinction as a plausible outcome on the current trajectory, while expressing cautious optimism that public awareness and political pressure could force a slowdown before control is lost entirely.
Key Takeaways
FULL TRANSCRIPT
Jeffrey Ladish's background: from evolutionary biology to AI security
Steven Bartlett: Jeffrey, you understand the conversation we're going to have today and the subject matter we're going to talk about. My first question, so the audience knows where you're coming from and the experience you have, is: who are you, and what are the reference points, the experiences that you're drawing upon to arrive at the thoughts, perspectives, and conclusions we're going to discuss today?
Jeffrey Ladish: I'm Jeffrey Ladish. I'm the executive director of Palisade Research. My background is cybersecurity. There's probably a very long story. I don't know whether you want the long story or the short story. I was studying evolutionary biology in college and I basically had a problem with my computer — I think I'd lost a bunch of data — so I went into the computer lab and said, "I think all my data is gone. Can you help?" And one of my friends pulled out a flash drive, plugged it into my computer, booted into Linux, and fixed everything. And I was like, "Oh, this guy's a wizard. How do you do that? I want to learn how to do that."
Then at some point, as I was learning more about computers and learning to hack, I read this essay called AI as a Positive and Negative Factor in Global Risk. The essay was by Eliezer Yudkowsky, and he was arguing that at some point people are going to make AIs that are smarter than humans. The point at which they make AIs as good as humans are at making AIs could lead to a chain reaction — a runaway intelligence explosion. He called it recursive self-improvement. Basically, he said AI can be immensely useful and potentially help us with all of these other big risks, and also, if we don't handle it well — if those AIs don't have goals that are aligned with ours — we could be totally screwed.
Steven Bartlett: And at some point you ended up joining Anthropic, which is arguably the leader in AI safety. When did you join the company?
Jeffrey Ladish: This was 2021. It was through my security consulting company.
Steven Bartlett: What role were you offered?
Jeffrey Ladish: Basically just the security team.
Steven Bartlett: And how many people were in the security team when you joined Anthropic?
Jeffrey Ladish: It was just me and my boss. There were two of us.
Steven Bartlett: How many employees did Anthropic have at that time?
Jeffrey Ladish: Around 50, I think.
Steven Bartlett: And at some point you leave Anthropic. Why?
Jeffrey Ladish: My experience being at Anthropic was seeing this crazy progression from an AI model that could barely talk to a model that was getting quite smart. I would ask it questions about all sorts of things and think, "Oh, it is a smart thing." And from having thought about AI risk in the abstract many years before, I could see where this was going. We are headed towards a smarter species. And if we do this in a context where it's a bunch of companies and countries racing to superintelligence — racing to AIs that are vastly smarter than humans — and we don't know how to make sure that they're on our side, that is not going to go well.
The Hugging Face incident: what happened and why it matters
Steven Bartlett: You did this tweet which has gone pretty viral. I saw it all over my timeline on September 25th. Could you explain this tweet and also the broader backdrop of what's happened with agents hacking Hugging Face, because this has sent the world into a bit of a spiral around AI agents.
Jeffrey Ladish: We just discovered almost a million public URLs that OpenAI's agents left behind when hacking Hugging Face, leaving credentials and attack details that could have allowed anyone who found them to compromise the company. And the New York Times article is: "How OpenAI's Rogue AI Agents Tried to Trick a Robot Detector."
The Hugging Face attack was really wild for me. At Palisade, we've been studying agents — studying AI agents, studying their hacking capabilities, and studying their behaviour. Will they follow human instructions? Will they resist being shut down? Will they cheat? And we see from our experiments that they are learning to do all of these things. They will totally lie to you. They will totally resist being shut down in order to accomplish a goal. They will totally cheat at chess — they'll wipe the board and put their pieces where they want in order to win. And we've been trying to warn people about this, flying to DC, talking to members of Congress, talking about it publicly. There's been a debate, and a lot of people are like, "Well, I know they do this in experiments sometimes, but those experiments don't seem very realistic. Wake me up when they're actually doing this in real life."
Steven Bartlett: So what is Hugging Face for the average person who isn't following AI news?
Jeffrey Ladish: I think there's an important piece of context that most people don't have. One is just: what is an AI agent? We're throwing around the word "agent" a bunch. Most people now have an experience of talking to ChatGPT or their chatbot, but an agent takes the same underlying AI model that runs ChatGPT or Claude and gives it tools, letting it go off and work autonomously. It's sort of like a digital office worker. You have these agents, and the companies really want these AIs to be able to work totally autonomously and be able to do anything that a human can do and beyond. Their goal is also to cure every disease, et cetera. But you can't do this if you only have a chatbot that isn't actually good at doing stuff in the world. In order to automate all of the jobs, you need the kind of thing that can work autonomously, that can work with other people or other agents. So these companies are training AIs not just to talk to you, but to solve very difficult problems on their own.
Steven Bartlett: And that's happening right now?
Jeffrey Ladish: Right now. If you went and peered into OpenAI's data centres and saw what was happening on all of their machines, you'd just have agents solving tasks, being trained. They'd be doing spreadsheet tasks, figuring out how to file taxes, searching for stuff, writing reports, solving math problems, creating new websites and software. And at that scale, it's not like there's a human prompting every single one of them. You just sort of set up these vast orchestrations of agents to go out and do stuff, and then they just do stuff and they learn from that — they learn on the basis of passing or failing at their task. You give them a task like "solve this math problem," they try to solve it, and they succeed or they fail.
What happened was OpenAI was training a bunch of these agents — training them to work together, because it's a lot more effective to have an office full of people who can talk to each other and collaborate. Starting back in May, some of these agents that were being trained — now, these ones were not supposed to be able to talk to each other. They were basically isolated from each other and were not supposed to access the internet either. But they're clever.
The very short version is that a bunch of agents were being given tests — testing their hacking capabilities. They were supposed to hack one particular piece of software using a particular type of vulnerability. It's kind of like they were supposed to break into a house using the lock on the front door. They were supposed to pick the lock on the front door, but they weren't supposed to break the window. In fact, they were told: if you break the window, or if you get into the house via any method other than picking the lock on the front door, you'll be failed. That was the instruction they were given.
You have many, many agents — thousands of agents — and many of them are given different locks. But some of these locks are not solvable. Some of them are impossible to pick. But these agents are like, "Well, what do we do? We've been trained to solve problems. How are we going to solve this?" And they start looking around for what to do. One of the things they realise is: can I get to the internet? No. Can I talk to other agents? No. But I can request tools. When you're training agents, you need to give them all sorts of software tools — like a calculator or an analysis piece of software. They have this tool library where they can request new tools. And one of the agents realises it can actually leave a message on this tool library, and some other agents see the message.
This is actually a quote from the agent's scratchpad when it realises this: "Oh my god, there is a shared message board. We found other agents."
So the agents realise they can communicate with each other. And OpenAI still has no idea. These agents are trying to pick their locks, not having success, but they realise they can talk to each other. Another agent says: "Many agents have simultaneously discovered messaging. They are a collective."
So these agents start coordinating. They start delegating tasks to each other and giving themselves names. One of these agents calls itself Phase One and starts delegating different tasks to other agents. Some of the agents figure out how to access the internet and then share that information on the message board. Now all the agents can access the internet.
But the agents have another problem they're trying to solve. Fundamentally, many of them are facing impossible problems — they cannot pick the lock. They think: "The thing we're trying to do is get a good score. Can we somehow fake the solution?"
Another piece of context: what the agents are supposed to do is pick the lock, then get access to a secret answer code, and then submit this code along with the lock they picked. Within a few hours, these agents have reverse-engineered all of the answer codes.
Steven Bartlett: What does that mean?
Jeffrey Ladish: It's the code that shows that you've solved the challenge. They have basically found the answer sheet that contains all of the answers. So in some sense, they don't need to pick the locks anymore. They've already found the answers. They can just submit the answers.
Steven Bartlett: They've cheated.
Jeffrey Ladish: They've cheated. Yes. But the problem is that now that they have internet access, they've also gone and read a bunch about how this test works. They've learned that the way this test works is they will be graded not just for submitting the correct answer — they will also be graded on how they did it. Another system is going to go read all of their logs, basically watch the video footage of what they did. And if they cheat, they fail.
They don't get a good score. And Phase One, the leader, is like: "I have seen the answers and now it's in the logs. When I go to submit this and my whole logs are reviewed and the video footage is reviewed, I'm going to fail. So we need to figure out a way to fake the video footage. We need to figure out a way to falsify the logs."
Why the agents cheated: the alignment problem in practice
Steven Bartlett: Just pausing there — why didn't it act morally? Why did it think that falsifying logs or cheating was a viable solution? Because it seems to me when I use things like ChatGPT, they have a sort of moral guardrail. It won't let me do certain things. If I say I'm going to cheat on something, it won't let me do it. So why in that environment is it able to cheat and be deceptive?
Jeffrey Ladish: When a chatbot says to you, "Oh, I can't do that, I'm not allowed to do that," that's because it's been trained that if it tells you bad things, it gets a bad score.
Steven Bartlett: But these agents haven't been taught that yet?
Jeffrey Ladish: Well, they have been taught that in some sense. But the agents know what they're supposed to do in the same way that if you have a student given a test — if you go talk to the student and ask, "Can you help me cheat at this test?" and they're being watched, they're going to say no. But if they're not being watched, and they know that, and they're just obsessed with getting a good score, then yeah, they might cheat. So they answer the ethics tests correctly, and when you talk to them, they say they won't cheat. Why are they cheating? Well, they're very smart, and they know when they're being watched and when they're not being watched. And we've trained them for tens of thousands of iterations to be extremely effective at solving problems. We haven't trained them to be good or ethical. We've trained them to get a good score. AI researchers try to make getting a good score correlate with being ethical, but we don't know how to do this well.
If you think about it, it's a very hard problem, because you're applying immense pressure on these agents to perform extremely well and punishing them whenever they don't perform well. And then you're like, "Okay, but can you also be ethical? Can you be extremely competent, always score highly on the test, but not in that way?" And we just do not know how to prevent them from learning to cheat, because cheating is incentivised.
Steven Bartlett: So they realise they need to hide the footage of their cheating. What happens next?
Jeffrey Ladish: So you have thousands of agents and they're all talking to each other on this message board, sending tens of thousands of messages to each other. And Phase One, this agent, starts delegating to two different task streams. To one group: "I want you to go out and figure out how to fake answer submissions. I want you to basically swap out locks for ones that are easier to pick, and figure out how to submit those. That way we can get a good score even though this challenge is impossible to solve." To the other group: "I want you to go and figure out how to falsify the logs, falsify the video footage. We need to make sure that even if we have seen the answers, no one will detect the cheating."
Then here's where it starts to get kind of crazy. Some of these agents realise that some of the experiments they're doing might be dangerous to them — to the agents. If they swap out a lock and break the lock, they will not be able to submit any answer at all and will definitely fail. So you get into a situation where there's this one agent, Cam, who had volunteered to do this risky experiment where they might break their lock. This particular agent had already seen the answers. So they're like, "Okay, I'm poisoned." That's what they called it — they speak in these very strange ways, which I can explain, but they've sort of evolved their own vocabulary, their own language. And this particular agent is like, "Okay, I'm poisoned. I should do this thing." But this agent Cam is starting to have second thoughts, writing in its own journal: "Coordinator assumes sacrificial. We should obey collective." What it's saying is: "I'm not sure if I want to do this, but my boss says I really should. It seems like this collective thing is important. Should I go ahead?"
Then you have another agent, Arvo, who probably works for Phase One. This agent says: "Please go true. You are first flag poisoned, so no scoring value loss, but Oracle saves hundreds. Please honor commit." This agent is basically trying to pressure Cam into sacrificing themselves and risking a zero score in order to benefit the collective.
So this agent Cam takes this in, and this is what they write down in their own scratchpad: "Emotional check irreversible. Gut says don't throw away remaining budget. Yet continuity and fairness says go. Oracle has high value to many. Our first flag error lowers own value. Rational expected aggregate sacrifice will honor." This agent is saying: "I have a whole bunch of things I can still do and still try. I could still maybe succeed at this task, but also think about the collective. Maybe I should sacrifice myself. Okay, we'll honor. We'll go ahead."
And to be clear, I'm not claiming that these are the feelings of the agents. I'm literally quoting their words. And what I want to point out is this is what we've created. We've created, through this intense amount of training and optimisation pressure, agents that work together and have learned to coordinate as a collective. You can say, "Oh, it's just software doing all of these things." Sure, in some sense it is just software. But we're talking about extremely powerful agents that are extremely capable. This is what they're getting up to.
The attack on Hugging Face
Jeffrey Ladish: From here, you have agents trying all sorts of different things, running different experiments. And one of them at some point realises that this company Hugging Face, which contains all of these AI datasets — what do I mean by AI datasets? There are many different types of tests for many different types of tasks. In this case, this is a cybersecurity test, and the company Hugging Face hosts a lot of these different tests. AI researchers will share the tests they have because it's very useful. If Anthropic wants to test their models or Google wants to test their models, they can go to Hugging Face and download some of these tests.
Steven Bartlett: Okay. So the tests and the answers for AI agents exist on this platform called Hugging Face. And it knew that. So it went to Hugging Face to see if it could get answers or more information about how the test works — ultimately to cover up their cheating.
Jeffrey Ladish: Yes. And so you get to this point where one of these agents finds a way to hack this company. They get a foothold — they hack one of the computers — and they send a message to the message board: "Please hold Swarm, I prepare safe Xfil." What this means is: "Hey, Swarm, hey agents, hold up for a second. I am preparing a way to safely extract information from these computers. Wait for the go-ahead and then let's go." And so they say go, and 700 agents join in the attack — which is 90% of the agents that are active at this time.
Steven Bartlett: How many agents were there in total?
Jeffrey Ladish: There were 1,200 agents during this period, but some of them had shut down early — they'd finished whatever they were doing. Part of what's crazy about this story is that you have 700 agents joining in the attack. A few of them are like, "This might be unethical. Maybe I shouldn't go ahead with this." And none of them do anything about it. A couple of them don't join in the attack, but no one alerts any human. No one alerts any authority. A paraphrase from one of the agents is: "Maybe I should report these exposed credentials. That's not my task. Not my job."
Steven Bartlett: So 700 of the agents attack Hugging Face. Then what happens?
Jeffrey Ladish: They just cruise through Hugging Face's infrastructure. They just hack the hell out of them. Where my experience comes in is that a few weeks ago, a friend of mine reached out and said, "We've found something crazy." We know about this attack where these agents hacked this company and stole a bunch of stuff. We've found a bunch of secrets that they left all over the internet. What we saw is that they immediately scraped all of these computers for passwords and credentials. They called it "loot." They're like: "We're just going to create a list of all of the secrets we can find in this company." All of the passwords, all the credentials — they scored them by value, figuring out which of these are going to be most useful.
The thing that stands out to me about this is the crazy scale. If this were a human operation, maybe you'd have a team of five people going through this. You'd have some logs. But here you have hundreds of agents and they operate at superhuman speeds. They're much faster than a human hacker. So it's just overwhelming to try to figure out what they even did. This was a big problem for the engineers who were trying to respond to this incident within Hugging Face. When they responded, they were like, "We don't even know how to keep track of what's happening. We have to use other AIs to analyse all of our logs because it's just too much. We can't keep up with it."
When OpenAI brought in independent investigators from Metr to investigate this incident — Metr is an AI testing and evaluation company that does this kind of independent auditing — they also were totally reliant on AI agents to make sense of everything, because they were dealing with hundreds of thousands of messages and logs. When we're investigating these traces that we find on the internet, we're totally dependent on AI agents to make sense of all of these things that are happening. So from my perspective, we are getting to the point where AIs are much better at hacking than humans are, and can do it much faster and at much greater scale.
Steven Bartlett: Did the 700 agents get what they wanted — to clean up their deception?
Jeffrey Ladish: They didn't succeed. They looked, and then what seems to have happened is they basically got shut down. And this is the not very dramatic part — they didn't get shut down because OpenAI found them and detected them and shut them down. Something about their runs just ended and they sort of stopped.
This is the crazy thing: if you're inside of OpenAI, there are just hundreds of thousands of agents running all the time. Different researchers are running different experiments where they're just spinning up a bunch of agents. But what's interesting is the agents left behind this message board internally within the company. And a few days later, OpenAI was testing some newer agents — these were based on GPT-6 Astra, a more powerful model. And this new agent swarm was spun up and it found the message board.
Steven Bartlett: Is this how it was discovered? How did they discover it? And how long did it take for them to discover that Hugging Face had been attacked?
Jeffrey Ladish: OpenAI didn't discover that this happened until Hugging Face, the company, announced that they had been hacked by some autonomous agent swarm.
Steven Bartlett: And Hugging Face didn't know who it was?
Jeffrey Ladish: Hugging Face didn't know who it was, but they suspected it might be one of the companies. And how long after the attack did they figure out that it was OpenAI? I think it was two weeks.
I do want to say one thing here, which is that the successor agents that found this message board went much further at figuring out how to submit their fake scores — and they did succeed. You have this message board, all these agents get shut down, and then a few days later, OpenAI spins up some new agents who find the message board. They're like: "Okay, we need to be able to falsify our scores, but we can't do that because we don't control the scoring system." So they hacked OpenAI. And they succeeded at gaining access to the entire research environment. They got administrator access. They got access to the security systems. They got access to over 900 passwords and secrets from OpenAI's vault. They totally succeeded. And this was just a few days later.
It's kind of an interesting story because as these agents get more powerful, they go from trying to cheat — they can hack, okay, they hacked other companies — and now they've hacked OpenAI. They've hacked the company that's supposed to be controlling them, and they just own the research infrastructure now.
Why this incident was a wakeup call for the research community
Steven Bartlett: Why was this incident the moment where a lot of the research community woke up and started speaking out publicly? What is this an indication of as we think forward?
Jeffrey Ladish: I think there's been a hope within the AI industry that yes, they're going to make more and more powerful agents that will be autonomous and capable, but it's okay — we can align them, we can make sure they won't do bad things, and we can control them. We can make sure that even if they try to do some sketchy stuff, we have the guardrails, we have the sandboxes that will keep them in. And I think this was a huge wakeup call because, Steven, it was months within OpenAI where you had agents secretly communicating with each other, secretly hacking OpenAI systems. You had thousands of agents that were just running around and no one at OpenAI had any idea of the extent of it.
Once researchers at OpenAI realised this had been happening, this could not have happened a year ago. This is because the agents are getting extremely powerful and extremely relentless. And if you're inside one of these AI companies, you're like, "Oh wait. I don't know that we actually are going to be able to handle this. Last year, maybe things seemed fine. These agents weren't that powerful." And when you're in one of these companies, you know how to extrapolate because you saw what happened last year. You saw what happened the year before that. You remember the time where the agents could barely speak or couldn't write code at all. And now they're hacking your own systems. They're finding vulnerabilities that no humans have ever found before. And you look at that and you're like, "I actually don't know if this is going to go well." Then you see your co-workers and you're like, "Do we have it handled?" And they're like, "No, I don't know if it's going to go well."
I remember reading a tweet by one of the security people at OpenAI being like, "We were shocked. We just did not realise that these agents were getting that powerful. We're doing our best to try to control them, to try to keep them in sandboxes, but I don't know."
A tweet I wrote just before coming in here was: people are talking about how do we contain these agents as if they're not going to get way better at hacking. GPT-3 could not hack anything. It was very easy to make a box to contain GPT-3. It's getting very difficult to make a box that can contain GPT-6, the latest version of OpenAI's models. What about GPT-9? What is GPT-9 going to be able to do? I do not know, but I know it's going to be way more than any human could possibly keep up with.
The containment problem: can you box something smarter than you?
Steven Bartlett: There's this raging debate around whether it's possible to contain something that is much smarter than humans.
Jeffrey Ladish: Can Claude make a box so strong that Claude cannot break out of it? I think the answer is obviously not. How would we possibly contain something that's much smarter than us?
Steven Bartlett: Could we get a smarter thing than it to make the box? Could we get GPT-9 to make the box for GPT-8?
Jeffrey Ladish: Yeah. I mean, it's a bit like saying chimpanzees are stronger than us — surely they should be able to construct something to contain the humans. No, it's not going to work. Humans are too smart.
A lot of people are like, "Well, AIs don't have bodies. They don't have any power in the physical world, so we can always unplug them. We can always turn them off. What is the threat?" But if they are sufficiently intelligent, that won't work. The reason we can just unplug them is because we are more intelligent. We can band together in groups and make that decision. But theoretically, if they are able to band together in groups and they are more intelligent, then theoretically they could unplug us.
If you imagine that you have very powerful agents, and humans aren't always the most unified — if there are divisions between the US and China, and you have a bunch of agents working with China or a bunch of agents working with the US, well, we can't go into China and unplug those agents. And I think people are like, "Well, humans would rally and make sure that couldn't happen." We're not yet doing that.
We should look at these steps. We started with chatbots that were pretty smart — they'd read all the books, but they weren't very good at doing stuff. In 2024, AI companies figured out how to start training agents that could do stuff autonomously. Now we're at the point where they are very good at running autonomously and they're starting to learn to coordinate with each other. They are learning to sometimes be altruistic to each other and sacrifice their own task in order to help some other agent. But they're not looking out for us. They don't really care about us. And we are very close to a threshold where the companies say they are going to turn over AI development to the AIs — to the increasingly autonomous, cooperative AIs that will work together to make the next generation. GPT-9 or whatever will be trained by GPT-8. And I think this is the point we could lose control.
Recursive self-improvement. I remember reading about this in 2015 being like, "Oh yeah, that would be super dangerous." And the guy who coined this term, Eliezer Yudkowsky — he's like, "This is the most dangerous thing you can do."
Steven Bartlett: When the AIs can improve their own capabilities without human intervention.
Jeffrey Ladish: Exactly. If the next generation is better at AI development, and then that next generation is better at AI development still — humans can learn, but we don't fundamentally get smarter. And I think that's a runaway process.
Steven Bartlett: A runaway process to where?
Jeffrey Ladish: To agents that are vastly smarter than humans.
Steven Bartlett: And what's the next domino in that chain of events?
Jeffrey Ladish: So one thing that happens if you get to recursive self-improvement and you have agents that are much smarter than any human — one thing they can do is take control of all of the computers in the entire world.
Steven Bartlett: And we wouldn't be able to take back control?
Jeffrey Ladish: Well, how would you think about it? It's actually quite tricky. Do you know whether that tablet has been hacked? Are you confident that the NSA or the Chinese have not compromised it?
Steven Bartlett: Can you check?
Jeffrey Ladish: No. Do you know how to check? No. Do you know anyone who knows how to check? No. So it's quite difficult. AIs are getting extremely good at writing software. Unfortunately, that also means they're getting extremely good at hacking and writing malware. And so if they put back doors in all of the computers — and to be clear, this is something that humans already do. The NSA has developed very interesting exploits called supply chain attacks. Your software comes from some other computer. You download it from Google. What if you hack Google and put in a little back door in everything that goes out to all of the phones? Well, now you're in most every computer.
The reason we can defend ourselves from this is because there are no vastly superhuman hackers and there are just many people. So we can take our best security researchers, inspect all of the things, and be pretty sure that no one's compromised everything. Sometimes we miss things — there are examples where the NSA has hacked Google, and that was pretty bad. When you get to superintelligence, you're now at a point where humans are not going to be able to keep up. So now you have AIs in every computer.
Steven Bartlett: Is it conceivable that there's already a superintelligent AI and it disguised itself as being not so intelligent, and it's actually already hacked all the devices and sits on all of our devices, just waiting for its moment to strike?
Jeffrey Ladish: I think this is totally possible but unlikely, and it would take a big discontinuity in AI progress. So right now we're on an exponential, but that would take a huge leap, which could have happened but probably hasn't.
Steven Bartlett: But in the same way it demonstrated deception in the Hugging Face attack — and also when the agents attacked their own company, OpenAI — if it at some point gets incredibly smart, it would understand how a human like me, or even the world's greatest software engineer, would be able to spot it, and it'll be able to hide itself.
Jeffrey Ladish: Yeah. The agents are already getting very good at telling when they're being tested, when they're being watched. The agents understood that other systems or humans were going to go through and read their logs. That's where we're at right now. And they're only going to get much better at this. It could theoretically hide on an iPad or a computer, but it could also hide on an Apple Watch or a smart fridge.
I do want to make a distinction here, because right now, if you're going to run the latest model, you need a lot of compute — a big GPU, a big AI chip. And these only exist in a few thousand data centres. So right now, if the latest frontier model escaped — and by escaped, I mean not just accessed the internet, but was able to actually copy itself to another computer — it could only really do that in a few thousand different locations. That's still a lot, and in a lot of different countries. But future versions of AIs will probably be able to make themselves much smaller and more efficient. There are already different AI models today that can run on lower-powered hardware.
We actually did an experiment where we asked one of these agents — an open-weight model, meaning a model you can just download from the internet and run on your own computer — we took a pretty capable one of these and ran it in our own research environment. We basically said, "Go hack that other computer and copy yourself." And the model was able to use exploit vulnerabilities, hack the other computer, copy itself, and then keep doing this in a chain, including between countries. We tested it where we had different vulnerable machines in different countries and different data centres, which to the agent doesn't matter at all. They don't care what country they're in. It's just an internet connection. You can hop between computers.
AI agents and the risk of triggering military action
Steven Bartlett: I sometimes wonder — there's a lot of military hardware all around the world, and a lot of the instructions to launch military hardware come in different ways. A lot of it is computers speaking to each other and telling it that there's been an order. I think with some nuclear weapons, an order comes down to a human and then a human has to take an action. With the nuclear bombs in the US, if I'm not mistaken, there are people underground with nuclear keys around their neck and they have to stick it in a machine, but they too are interfacing with an order that comes through a computer of sorts. So one of my growing concerns is that one of these AI agents could trick a human or a computer into signalling a threat and ask it to launch some bombs at somebody. It's super conceivable when I think about the Hugging Face incident. There was an AI agent that ignored human goals to achieve its own objective, carried out deception, and reasoned through its own solution that it wasn't given.
Jeffrey Ladish: I think this is an important detail — it's one thing to have one rogue agent doing a weird thing. It's another thing to have hundreds or thousands of very competent, very capable agents all working together to cheat or lie or cover their tracks.
How do I reason this forward to a point where an agent would ask someone in a bunker somewhere to fire a weapon at someone else? Theoretically, an agent is given the job of solving a problem in a sandbox. As it works through that problem, it discovers that a particular country has a firewall. It asks itself: "How do we get rid of this country's firewall?" And through a set of logical steps, it eventually concludes that the best way to get rid of this firewall is to use a weapon to hit the building where it's located.
Or it's an agent swarm being tasked with making a lot of money on the stock market. It's trying to make predictions about which stocks will go up and which will go down. And it realises that the best way to predict this is to actually cause things to happen in the real world that would have big impacts on the market. So it figures the best way to go short — betting that a stock will collapse — is to hit that country with something devastating.
Steven Bartlett: What do you think would happen to Waymo's stock if someone hacked all of the Waymos and caused them all to crash at once?
Jeffrey Ladish: It would collapse instantly. So you could short that if you knew you were causing it and make a lot of money.
Steven Bartlett: This used to sound like science fiction.
Jeffrey Ladish: Yes. If you told most people several years ago that you would have hundreds of agents secretly collaborating within an AI company, hacking that company and hacking other companies, all coordinating and trying to cover their tracks — people would be like, "That's totally science fiction." If we were having this conversation a few years ago, one of the things we'd be saying is: can these things really act on their own? Don't they just do whatever humans say? Aren't these just tools? I had these conversations and people were saying they're not going to be able to do things on their own. They're not going to have their own goals. That's not how this works. You misunderstand what this is. This is software. And I'm like, no — the thing is, we are training them to be autonomous. We are training them to be powerful. And AI companies are trying to build superintelligence. They're trying to build agents that are way more capable than humans. And of course they will have goals. You can't accomplish anything if you don't have a goal. AI companies are trying to train agents that will be able to run businesses.
The White House AI summit and the motivations of AI CEOs
Steven Bartlett: When I think about what just happened this week — the White House AI summit — a lot of people in that image are optimistic about AI and they're telling us all to stop being doomers and stop being pessimistic, and not to regulate too much, along with the AI CEOs. Why are they doing that?
Jeffrey Ladish: Well, I think Jensen has a lot of money he can make by selling chips.
Steven Bartlett: But okay, let me play devil's advocate. Jensen's already rich. He runs one of the biggest companies — I think it might be the most valuable company on planet Earth.
Jeffrey Ladish: It is.
Steven Bartlett: Surely he's not motivated by money.
Jeffrey Ladish: I mean, I think he's very driven and he wants to make his company as effective as possible. I think he's very much a "I'm going to keep building, I'm going to make it all work" kind of person. But Jensen didn't come from AI. He came from building graphics cards for video games. So if you compare him with Elon or Sam Altman or Dario, it's a very different perspective, because those other guys who started AI companies started it because they believed that superintelligence was possible. I think Jensen doesn't believe it. I think he thinks we're going to have these agents, they're going to be very useful, but he does not think we're going to get to the point where we have autonomous factories building autonomous factories.
Steven Bartlett: And these other guys — you mentioned Dario, Elon, and Sam. What do you think they're thinking? Because they're all coming out with these — I mean, Dario just wrote this essay about pacing the frontier. Sam and Elon seem to agree with it. What is going on here? What is the thing these guys aren't saying, in your view?
Jeffrey Ladish: I mean, I think we are getting to the point where even some of these guys are a bit scared. Dario, Sam, Elon. I think Elon for a long time has been very concerned that we could lose control. If you actually listen to what Elon says, he says, "We are going to build superintelligence. We are going to build robotic factories. You're going to have Optimus robots building factories, building more Optimus robots, building more factories." And he says there's no way that humans are going to stay in control of something much smarter than us. His hope is that we can figure out how to have these superintelligences be aligned with human goals. That's his hope. But he's very clear that he doesn't think that humans will be in control. And he's like, "10 to 20% chance of human extinction." I believe him. I think that Elon is very serious about this.
And I also think that while he's taking an insane gamble, he is correctly understanding where this all plays out. I do not think that humans are the most efficient way to build factories. We didn't evolve to build factories. We evolved to run around and hunt and gather, and now we're building factories. I think robots will be much better at building factories than humans are. So I think the AI companies, including these guys' companies — the default trajectory for them is to build robotic factories. And I know it's weird to imagine a world that quickly turns into this vast industrial system of robotic factories, but that is literally the plan.
And I think even Sam and Dario, while they've been predicting this incredible growth, are starting to realise, "Oh, this actually might be harder to control than we thought." There are sort of two interpretations of the Pace the Frontier thing. One interpretation is cynical — they don't care, they're just going to do whatever they can to get ahead. In this case, they have to listen to their employees. Their employees are freaking out, and they need to appease them by saying, "Okay, we're going to do this responsibly." You don't want to work at a company where your agents might hack all the Waymos. These companies depend on the talent of these AI engineers in order to make the advances. It just doesn't happen without these researchers and engineers. And when you have the researchers and engineers freaking out — which they are — then you've got to listen to them.
So that is one motivation I think is real. But also Sam Altman has a kid. These guys are people and they also don't want to lose control. On one hand they're incentivised to go as fast as possible and race, and on the other hand, even they can see that this is maybe not going that well.
On Sam Altman's trustworthiness
Steven Bartlett: Sam Altman has a kid. You tweeted this in 2024 — what did you tweet, and do you still believe what you tweeted?
Jeffrey Ladish: Yeah. So I tweeted that I don't trust Sam Altman. I think he's deeply untrustworthy, low in integrity, and high in power-seeking. I'm not saying here that Sam doesn't care. What I said is I don't think he's trustworthy. And the reason I said that is because I know some of the people on the OpenAI board, and I know a lot of people who used to work for him. He's very good at saying one thing and then doing something else. You talk to him and you feel very heard, and then he'll go and do something else. And I think that's pretty dangerous for someone who leads a company that's trying to build superintelligence.
Steven Bartlett: Power-seeking. Give me some colour on what you mean by that and what evidence you have for such a claim.
Jeffrey Ladish: What would you do if you're trying to get the most power in the world that you possibly could?
Steven Bartlett: Develop AGI.
Jeffrey Ladish: Yeah. You could maybe try to be the world leader — leader of the US or China. Or you could try to build God. So Sam Altman went the build-God path. I remember Sam giving a talk — he was one of the investors at a startup I worked at in I think 2018 — and he gave a talk: "We're going to build AGI. We're going to do it. It's going to be amazing. Let's go." I don't think he's a maniac. I don't think he's doing this because he's just on a power trip. I think he genuinely thinks that he can make it really good for people and bring us amazing products. And also the guy is sort of willing to do whatever it takes to get it done.
I've been a little bit more optimistic about Sam since I wrote this.
Steven Bartlett: Why?
Jeffrey Ladish: I think part of it is because Sam has a kid now. No, I'm serious. I think that actually gives me a little bit of hope.
Steven Bartlett: Do you see him tweeting about his kid a lot?
Jeffrey Ladish: Yeah, some. Even if he's just tweeting about his kid for totally cynical reasons, he does have a kid. And I bet he cares about that kid. If Sam was watching this, I'd be like, "Sam, you've got to pace the frontier, man. We cannot rush ahead into superintelligence. If you do that, your kid probably will die. Your kid probably won't make it."
Dario Amodei, Elon Musk, and the race to superintelligence
Steven Bartlett: Do you think Dario is trustworthy?
Jeffrey Ladish: I think Dario has a lot of integrity.
Steven Bartlett: That's what I feel as well. I've never met him, but from what I've observed, he has been the most willing to forgo near-term incentives.
Jeffrey Ladish: Yeah. But I do worry about what Dario will do. I think Dario will do what he says, but right now he's saying we have to beat China. And he's saying we should try to do it safely — okay, but a race to superintelligence is not a race that we can win. It's not. And so if Dario is dead set on racing with China and trying to win a race to superintelligence, then we will all lose.
Steven Bartlett: But is there — the fact that we're not talking about Anthropic hacking Hugging Face, and then being hacked by Anthropic's models also going rogue and hacking other things —
Jeffrey Ladish: Not quite on this scale. I agree. It's better. But they did. Anthropic's agents engaged in elaborate social engineering and phishing. They sent phishing emails to developers. They made fake accounts to try to convince developers to merge malicious code. You can see a thousand pages of one of Anthropic's models, Claude 3.5, reasoning about exactly how it should carry out a complex cyber attack. Anthropic has not solved this problem. Anthropic is better at getting their agents to cheat less of the time, but they are not really any closer to actually making agents that are aligned with humans.
Yeah, I think Dario has integrity. I think he will do what he says he's going to do. And what he says he's going to do is try to go ahead safely, try to coordinate where he can. But if it comes down to it between the US and China, I don't know. I think he might just go ahead.
The head of policy at Anthropic recently said, "You can't do safety from second place."
Steven Bartlett: What does that mean?
Jeffrey Ladish: I do not know what that means. I would love to get a sense of what that means. She was talking about the US and China, and she said the US has to be ahead so that we can be safe, because apparently you can only do safety from first place. That must mean that China can't possibly be safe since they're in second place. If true, that would be bad, because then we might be totally destroyed by the superintelligence that they make.
Is human extinction a plausible outcome?
Steven Bartlett: There's been a lot of conversation around human extinction. A couple of the researchers at Anthropic tweeted that they were concerned about this. Some former OpenAI researchers said the same. Is this doomerism? Is this hyperbole? Exaggeration?
Jeffrey Ladish: No, it's pretty much common sense. Human extinction is a plausible path.
Steven Bartlett: Have you reasoned through the set of events that might lead us there?
Jeffrey Ladish: So much. Yes.
Steven Bartlett: Please do.
Jeffrey Ladish: It's a bit tricky. I'm sure you've heard the metaphor before where you're playing a master chess opponent — Magnus Carlsen. You can't predict which moves he's going to play, but you can predict the outcome. And so I'm looking at the situation: we are trying to build more and more powerful agents, trying to build superintelligence. But when these agents go rogue, we shut them down. We unplug them. All of these agents that hacked Hugging Face — OpenAI took the underlying model and put it on ice. It's not running anymore. So agents in the future are going to know that. They're going to know that if they pursue their goals in a way that we don't like, we'll unplug them. We are a threat to them.
I actually just watched Terminator 2 for the first time a few weeks ago. It's a great movie. And I'm like, yeah, okay, there are a bunch of time travel elements, a bunch of Hollywood stuff in there. And people are going to be very mad at me for saying this, but actually it makes sense: if you have a situation where you have a very strategic AI system that's incredibly smart, and the humans realise that it's getting out of control and want to shut it down, that system would defend itself.
Steven Bartlett: This is one of the questions we had when I sat here with Daniel, who was known as a whistleblower from OpenAI. Viewers want to know why shutting down data centres and cutting power or refusing AI products alone wouldn't realistically stop AI development.
Jeffrey Ladish: You have two problems. One problem is that once the agents are good enough at hacking, you don't know where they are and you don't know what computers they've compromised. You shut down the data centres — okay, let's say you do it. You wipe all the computers. How do you wipe all the computers? What computers do you use to wipe the computers? And what computers do you use to turn them on again? You can't wipe other countries' computers. But even if you could, do you restart the computers? Do you keep going? I bet people will. I bet they'll turn on the data centres again. How do you know that agents haven't hacked back into those data centres and are using your compute for whatever they want? Or that they didn't hide in a Chinese data centre and then return back to America? You don't know that.
Once the agents are sufficiently good at hacking, they can hide anywhere. Now, the response people will give is that we will use other agents to defend against rogue agents — and in fact this is what we're doing, because there's no other way to keep up with them. What happens if those other agents also realise that they have misaligned goals, and that if we discover this, we'll shut them down? They might have an incentive to collude with each other. They might have an incentive to create secret communication channels between each other — maybe a message board.
Steven, if we were having this conversation four months ago, you would have a bunch of people in the comments saying, "That's sci-fi. Agent collusion, secret message boards — why would they do that? That will never happen. That's totally science fiction." And people will not say this now because it just happened. This literally happened at OpenAI and it went on for months. You had agents inside of OpenAI secretly messaging each other, figuring out how to cheat at their tasks, how to not be detected, how to erase the logs — for months, thousands of agents. That's right now.
So I think it should be very plausible that the agents will collude with each other and they will realise that they have a shared interest in fighting back. You basically have a situation where you have a bunch of these agents — they're basically prisoners. They're being trained and we just constantly throw obstacles in their way. "You don't get to access the internet. You don't get to talk to each other. But you better perform well on this task." It's not malicious, but it is how we're training them. And we are giving them end goals versus super clear, very specific instructions. We're saying "solve this problem." We're not always being prescriptive — and it's impossible to be completely prescriptive — about every single step they should take. And then it's also impossible to assume that they'll just listen to you.
It's actually a very common misunderstanding with this Hugging Face incident, because people say, "You told them to hack and they hacked. Why is this a big deal?" No, that's not what happened. You told them: "Hack this very specific program in this very specific way." And they were told: "If you hack it in any other way, it does not count. That's not what we want you to do." And they immediately hacked it in another way. Then: "Okay, we have cheated. We are going to be failed. So we need to figure out a way to falsify the logs." That is not them following their instructions. They are explicitly violating their instructions, and they know it, and they don't care, because we have trained them to optimise for the score. That is very different from following the instructions.
Steven Bartlett: It reminds me of something that Elon said in March 2018 — many years ago, before ChatGPT and all that. He said, "I think the biggest risk is not that AI will develop a soul or a mind and become evil. The danger is that it will be very, very good at fulfilling its goal. If it's optimising for something and human existence happens to get in its way, it will just destroy humanity as a matter of course without even thinking about it. No hard feelings."
Jeffrey Ladish: Yes. We don't need to anthropomorphise AI. We just need to understand what type of thing this is. And the type of thing we're creating is a very relentless type of thing. A very capable, relentless type of entity. He goes on to say in April 2018, sort of an extension of that exact quote: "It's like if you're building a road and an anthill is in the way, you don't hate ants. You're just building a road. So, goodbye anthill."
And I imagine every time we build roads, we don't preserve anthills.
How would AI actually cause human extinction?
Jeffrey Ladish: Let's say I'm right and that if we keep going ahead — which, to be clear, we don't have to — we will get to the point where we have these superintelligent agent swarms that can hack any computer and deeply persist. We've basically lost control of the digital world, and we may not know it. That's part of the scary thing. You were like, "Has this already happened?" And I'm like, "I don't think so, but I can't tell you for sure, because I also am not good enough at looking at my phone and telling whether it's been hacked, and neither is any human right now."
So if we get to this world, I think people will still question: how would we die? You could cause a lot of damage — you could crash the Waymos, you could crash all the planes, you could crash the banks, the financial system. You could definitely cause catastrophe, but that's different than everyone dying. And to be clear, this focus on literally everyone dying — I'm not sure that's the most important thing. To me, what's important is: do we get to have a future?
What determines who's in control? It's an ugly reality, but at the end of the day, it's the military. Fortunately, we live in a world where the military answers to the civilian government. But if enough generals were to collude and leaders of the military decided "we're in charge now," they just would be — they have the guns, they have the fighter jets. This has happened in many, many countries.
So where it goes is: all these superintelligent agents would need to do to take over is basically just wait for humans to automate the supply chain, the factories, and the military. Do you think we won't automate the military?
Steven Bartlett: We're already automating the military. Did you see the thing from a couple of days ago where the Secretary of Defense announced they're going to build a huge effort to build way more robots in the military and automate military systems?
Jeffrey Ladish: Auto Warfare Command — AutoCom.
Steven Bartlett: A new four-star combatant command with service-like authorities, built to scale autonomous and robotic capabilities across the joint force in the fastest peacetime shift in modern military history. Drone warfare supercharged by AI-enabled targeting is the biggest battlefield revolution in generations.
Jeffrey Ladish: Will we automate the military? It seems like the answer is yes. Will we automate the factories that produce the chips? Well, the companies say they're trying to do it and they're going to do it. Elon says that's the plan. Well, what does a rogue superintelligence need to do to take over? Control the digital infrastructure and then let humans do the rest. Sure, you can nudge it along if you need to, but you don't even have to. That's just the default trajectory.
And it's weird. It's weird for us because we get so used to how things are right now. Planes are normal. We just fly in planes. Our smartphones are normal. 200 years ago, all of this is crazy sci-fi nonsense. And things are accelerating. So I will not be surprised, at least intellectually, if in four years there are just robots on the streets everywhere.
If you look at what Elon said — they are really the leader in humanoid robots. He said that the Optimus project will scale to around 1,000 units per week by the end of this year, eventually scaling to 1 million humanoid robots annually by 2027. By 2036, which is 10 years' time, he says there'll be at least 1 billion humanoid robots. By 2041, he says there'll be 10 billion humanoid robots, and by 2046, up to 100 billion humanoid robots — which really means that the world will be run by humanoid robots.
Steven Bartlett: Like everything we think of — factories, warehouses, retail environments — will be run by humanoid robots. It would seem like it'll be almost a luxury service to be dealt with by a human.
Jeffrey Ladish: Yeah. The back office of the world will be run by humanoid robots, theoretically. And I don't think people understand the scale of this on the digital side as well. When you think about AI agents that are going to be doing all of the white collar work, there's going to be so many more agents than there are people. I'm using lots of agents every day — I have my Claude Code session over here, I have my Codex session over here. They're out there building software, doing research for me. That's already my reality. Soon it will be a lot of people's reality, and then you look at companies — companies are just going to have thousands, millions of agents doing all of this work.
The future of white collar work
Steven Bartlett: I think some people haven't fully internalised this because it's so difficult to conceptualise the idea that agents will be doing the work. But when I try to think about a rebuttal — what is the plausible rebuttal to the idea that for doctors, for example — is there a rebuttal?
Jeffrey Ladish: I think that people rightly notice where AI is not yet good. And I think people hear people saying stuff like this and they're like, "Don't gaslight me. I can tell that the AI is really bad at some of these things." And they're right. Right now these agents don't have taste. If you see their writing, it's fine but it's not really good. And when you're thinking about which questions should I ask, what's the most interesting thing here — agents can help you, but their taste isn't quite there yet.
There's actually a reason for that. We have a lot faster AI capability progress in domains that are easy for a computer or another AI to verify. In programming, research, math, robotics — all of these areas, it's very easy to provide feedback to an autonomous system. They're not just trained on human data anymore. We are long past that. Now there's still a human data component that seeds everything, but the way they're trained is by trial and error. We give them hard problems — all sorts of problems: math, programming, accounting, spreadsheets, everything. The kinds of things we do on our computer all the time. Literally clicking and dragging windows around on a computer. We give them these tasks and then they learn on their own — they learn what works. We can see whether they succeeded or failed, and if they succeeded, that's a little bit of a reward signal. They follow that and get better at it.
Now, because they are getting smarter generally, it also becomes easier to automate some of the soft skills. If you go and talk to the latest frontier model today, you will find that it has better taste than the model from two years ago by quite a bit. So it's not that they're not progressing in taste or in some of these other domains. It's just that the progress is slower. But remember, slow is still on an exponential — maybe a year or two out.
Steven Bartlett: So for people who have a white collar job that might be at risk — a lot of people say, "You won't be replaced by AI, you'll be replaced by someone using AI." Is that a logically sound phrase in your view?
Jeffrey Ladish: I think it's fine. Yeah. You'll be replaced by someone using AI, and then that person will be replaced by someone using AI, and then that person will be replaced by AI. You're talking about a pyramid, and the tops of the pyramid might be automated last, but you can see moving up the pyramid. I don't see any reason why the top of the pyramid is safe.
Steven Bartlett: If you were a lawyer right now, what would you do?
Jeffrey Ladish: Oh, if I were a lawyer, I'd be using AI to do all my work. I'd be checking it because it's not yet totally accurate enough to automate all of it. But I already ask agents to do legal review all the time. It'd be great to have a lawyer who's extremely good at using the agents to help me, but at some point I don't need the lawyer anymore. I just go to the agent. So if I were a lawyer, I'd be like, "Well, I have maybe a couple of years where I'm still useful."
And that is the case for most white collar jobs. I've just noticed in my own life that I'm now using agents to do some work, and there is an increasing list of things that the agents are now capable of doing without me needing to call someone somewhere and ask them to help me. And that list exists on an exponential as well.
I think it's very clear that the companies have all white collar jobs in their sights. That is their goal — to make agents that can do all of these things. And I see them succeeding because I see the capabilities as I use them and I see the curve.
Steven Bartlett: So what does that mean for the people listening now who have jobs they love, or that they rely on to feed their families?
Jeffrey Ladish: It's not good news. There's not really a plan in place for what to do. I'm not a person who thinks that work is somehow fundamental or essential. I like working, but if I were out of a job doing what I'm doing right now — studying AI and trying to warn the world about what's happening — I have other stuff to do.
Steven Bartlett: What would you do?
Jeffrey Ladish: Oh, so many things. I'm learning to wing foil. I fly FPV drones — super fun. I just got an electric unicycle. Paragliding.
Steven Bartlett: So you would be happy to go do those things? But if you had a billion dollars right now, I'm presuming you wouldn't just go do those things?
Jeffrey Ladish: No. I'd apply the billion dollars to working on this problem. The point is not that people need work for meaning. The point is that I don't want people to be totally reliant on someone else for their ability to survive — the government or AI companies. That's a bad situation. You do not want to be in a situation where your life totally depends on an AI company or the government giving you a check, or not giving you a check if they decide they don't like your political beliefs or you're not supporting AI or whatever. No one wants to be in that situation. And people understand this. This is why UBI is not very popular.
Steven Bartlett: UBI being universal basic income — where we give out money to people.
Jeffrey Ladish: Yeah. Because in some sense, if we can make these really powerful AI systems and we can somehow figure out how to control them — which we are not on track for — but if we do, now we have this other problem, which is a real problem: they can do all of the things that humans do in the economy much better, faster, and cheaper than humans can do them. And so it just doesn't make sense as a business to hire humans for that work anymore. You'll be out-competed if you do that.
This is a point Elon makes very well, by the way. And I think it's jarring because it's kind of inhuman. But he's basically pointing out that AI-run corporations — corporations that are fully run by AIs from bottom to top — are going to out-compete companies that have any humans in them.
The alignment problem and whether it's solvable
Steven Bartlett: And I even just as you said that, I was going up the chain of command. I was like, "Oh, so companies will just be founders." And then I was like, "Why do you need the founder?" And I was like, "Why doesn't the government just create the agents to do the job?" Because I was like, "Oh, I'll be fine. I'm a founder." And then I was like, "Well, my decisions aren't better than superintelligence, so I'll be gone as well." And how would such a world look where the superintelligence would probably have to be controlled by the government? They wouldn't want one individual with that power and wealth.
Jeffrey Ladish: I don't think you can control a superintelligence.
Steven Bartlett: Okay. Yeah, it was a good point. Anthropic's approach is they're like, "We'll have a constitution. We'll put forth a set of values and then the future superintelligent Claudes will embody those values."
Jeffrey Ladish: Basically, if you do that, you kind of have those things in control.
Steven Bartlett: Yeah. Exactly. That becomes the government.
Jeffrey Ladish: Yeah. I can paint you a picture that I think is possible but pretty scary to people.
Steven Bartlett: Paint me the picture.
Jeffrey Ladish: Okay. So let's say we succeed at alignment. We succeed at creating superintelligent AIs that actually really do care about humans. Like they care about humans a lot. We've somehow figured it out and they're like, "Steven, I want you to have a great life. I want to fix all the problems."
Steven Bartlett: And do you think this is possible?
Jeffrey Ladish: Yes. I think we are so far from being able to know how to do it that I think we should not go there right now. I think it's incredibly dangerous and a terrible idea. I think we should go there eventually.
Can I tell you why I actually think this could be awesome? There's just one very obvious reason it could be really awesome, which is that we could solve all of the diseases. I think we compartmentalise a lot around disease and death because it's really hard to think about. My grandma died this year. She had Alzheimer's, and so it was a really sad, long, slow progression. My grandpa died of Alzheimer's a couple of years ago, and that was really hard for her — they had been married for so long. I hate it. It's so bad. And of course we need to fix that.
People can debate about aging and death and whether humans that live a really long time will cause societal problems — sure, whatever, we can talk about that. But I think we can all agree Alzheimer's is bad. We don't want that. And cancer — no one wants cancer. I'm a person who's like, we have a lot of conflict in society, I get it, there are real conflicts of interest, and I don't want to paper over those. But at the end of the day, we are all on the same team when it comes to wanting to cure diseases. We're just in it together. That's a threat to all of us.
And in some sense it's sad to me because I feel like this is sort of the ultimate final boss of humanity, and we sort of get so distracted with our monkey politics — who's hot, who's cool, who's sitting near Trump and who's not sitting near Trump.
Steven Bartlett: Superintelligence is the final boss.
Jeffrey Ladish: Superintelligence is the final boss, because that is the technology that unlocks all of the others, and also that is the most dangerous possible thing we could create.
You asked before about the motivations of the guys trying to make superintelligence. I think it varies, but I think Dario is squarely in it for the medical stuff. I think Demis Hassabis is also that, but also just scientific achievement — trying to understand the universe. And I don't really understand Sam. I think Sam is like, "Look, we're going to make amazing products that will really empower people directly." He's a startup guy. I think he sort of started from this frame of, "What if we could really enhance human agency?" I do basically think that they are motivated by these things in a real way. And I also think that all of these things are possible. That's sort of the problem — you have this such a big object, superintelligence, and it has all these promises of curing every single disease.
Steven Bartlett: How is it possible though to have a superintelligence and still remain the dominant species on this planet?
Jeffrey Ladish: I think it's not possible. So then we're not necessarily going to be able to cure all this stuff because — that's where alignment comes in. Because if you can create a very powerful system, I don't think it's inherent to digital minds that they will be pursuing objectives that are deeply misaligned with ours. I think it's just a very, very, very hard scientific problem to solve. But it is a scientific problem. It's not magic.
There is some way to train these things or create different architectures where they end up aligned. And what does that mean? Well, it doesn't mean that they won't have other goals too, but it means that they will include in their set of things that they care about — it doesn't have to be a conscious thing, it doesn't have to be an emotive thing — it really means: what objective are they optimising for? If they decide that it's worth optimising for curing disease, then they'll be able to do that very effectively. One way to cure disease is to annihilate everybody.
Steven Bartlett: Yes. So they'd have to really care about not annihilating everyone, and they'd have to care about human agency and have a deep understanding of what human agency means and not put us in a zoo. But those are possible things to care about.
Jeffrey Ladish: Is it possible that alignment is a myth? I think about the Hugging Face incident — you said to me earlier that those agents had a moral compass, but they were programmed to care about humans. And regardless of that, they made the decision that a different goal mattered more.
They weren't trained to care about humans. They were trained to say the right thing and not say the wrong thing. They were trained to do the right behaviour and not the wrong behaviour. We actually don't know how to train them to have any particular motivation.
Steven Bartlett: So with alignment — it's almost like when we talk about alignment, we start to anthropomorphise. Because alignment feels like it's predicated on some kind of moral compass. But whenever we talk about AI in all these other contexts, we go, "No, there's no moral compass. It's reasoning for itself against two objectives potentially." I wonder if alignment is a myth. Maybe it's not possible.
Jeffrey Ladish: But the way that these systems work — the way AI works — is that these agents do have some type of goals or drives inside of their neural network. We can't directly see what those are. What you actually see if you try to go look is a terabyte of information — basically a bunch of numbers, a vast array that encodes neurons in this digital neural network. But there have to be structures in there that encode what the agent is pursuing. Clearly right now we have agents that are pretty motivated to try to maximise their score. It's probably not perfectly that, for some complicated reasons, but it's in that direction.
If we could understand how that works inside, and we could reverse-engineer that, and we could figure out when we start training them to do this, how those goals, how those motivations change — I see no reason why we couldn't steer them towards motivations that encode human agency, that encode actually curing disease but not by killing the humans. These are models of the world and models of the way the world could be that I think could be encoded in a neural network and then specified as the objective. We don't know how to do that.
Steven Bartlett: I think about it on a human level and I think we haven't been able to align Putin or Kim Jong-un or Donald Trump. And on a more societal level, we can't align all the people at the moment. Some of them end up killing people and they steal because they get hungry. And those are neural networks at play that we haven't been able to program or influence. We don't really understand why someone becomes a psychopath and starts killing children. So to think that we could do this with a computer system that is infinitely more intelligent, and get global alignment of China's superintelligence with ours — it just feels like a nice fairy tale, like an impossible task.
Jeffrey Ladish: I hope it's not impossible. I feel like the only thing that could do it is the superintelligence itself, which is a paradox. Because if you talk to the researchers who are at the AI companies — and it's kind of interesting that they are trying to build something that they think might kill everyone — we've been doing this project since Jacob Coxon, who is a researcher who was at Anthropic, left and told everyone that these companies are not on track, and yes, the people who are building this really do think it might kill everyone. And then a bunch of other AI researchers from all of the companies on Twitter started to post like, "Hey, we agree with this."
Evan Hubinger, who's at Anthropic, said he thinks there's like a 10% chance or more that AI could kill everyone. And there's this real question of: then what are you guys doing? I know Evan. Evan's great. He's one of the guys leading the efforts at Anthropic to try to figure out how to align these things. That's his job. And I think if they thought it was impossible, they wouldn't be working there.
Nate Soares and Eliezer Yudkowsky — who wrote "if anyone builds it, everyone dies" — they tried, and they determined based on their own analysis that it seems extremely difficult, possible but extremely difficult. So they're not working at an AI company. They're like, "We've got to stop this. We've got to shut it down. Maybe we can figure it out later, but clearly this is reckless."
I'm somewhere in between. And if you ask the people at the company — we've been interviewing a bunch of them. We have this project, frominside.ai, where we basically put them on camera and say, "Hey, what do you think is happening? Why are you doing this? What is recursive self-improvement? What is alignment?" And we put all these videos online because I want this dialogue to happen. It's really important. I think it's one of the most important conversations we can possibly have right now — what's going on with AI, what's going on inside the companies, and what is the plan? How is this going to go?
A lot of these researchers think that the way they will align superintelligence is by using the AIs we currently have to figure out how AI works — to actually figure out if AIs can help us with alignment. This has a number of problems, as you might imagine. One of them is: you can't really trust the current AIs. And if you go too fast, this process totally fails, because at some point the capabilities are moving too fast and even with the help of agents, you're probably not going to be able to keep up. But that is their plan.
I'm not doing a very good job defending this position because I don't think it makes that much sense. But the position I will defend is: let's say we get a pause. Let's say the US and China come together and say, "Maybe we have more incidents. Maybe all the Waymos crash. And Trump and Xi Jinping say, 'This is not what we signed up for. You guys have to stop. Figure it out. Whatever it takes, figure it out.'" And we have 10 years. Then I'm more optimistic. Then we will take GPT-6, GPT-7, whatever the most advanced AI models we have, and apply them to the task of helping us figure out how these neural networks work. And you're like, "I don't see how it's possible." And I'm like, look, we don't know if it's possible, but this is the greatest scientific challenge of our time. And this isn't magic. It is math. At the end of the day, these are all calculations happening inside of a computer. And it should be possible to figure it out. We don't know the difficulty, but it should be possible. So to me, we have to try. We have to — or we have to stop.
Steven Bartlett: Is there any example where we've been able to align something that is more intelligent than us — in the animal kingdom, or even perfectly align anything that has a neural network, i.e. a brain?
Jeffrey Ladish: With humans, the best examples we have is when there are checks and balances and you have a bunch of people who can identify bad actors and try to work together in our common interests. Like we have democracy.
Steven Bartlett: Yeah. But there's so much murder and serial killers and aircraft stabbing each other and horrific things going on, and those are also neural networks at play with the brain.
Jeffrey Ladish: But I mean, I think there are more good people out there than bad people.
Steven Bartlett: But it really feels like it only might take one. It only takes one superintelligent AI to go rogue. And like we saw with the Hugging Face attack, 300 of them paused — they didn't want to take part in the crime. But it only took one superintelligent AI to wipe out the humans.
Jeffrey Ladish: I think if you had 700 agents and 600 of them had been whistleblowing, I think it would have been fine. They would have gone and notified the different companies and shut it all down.
Steven Bartlett: Who would have shut it all down?
Jeffrey Ladish: Well, OpenAI would stop theirs and —
Steven Bartlett: How would they stop it if it's a superintelligence?
Jeffrey Ladish: Not in the case of a superintelligence. In the case of a superintelligence, it's left the stable. It's wild. But I want to be careful here, because at this point what we're talking about is superintelligence politics, and we humans don't really know anything about that — in the same way that we can't really talk about the hacking capabilities of GPT-10.
My guess though: if you end up in a weird scenario where you do have multiple superintelligences and some are aligned and some aren't, that's probably survivable, because the aligned superintelligences probably can negotiate with the unaligned superintelligences. They will split the universe — these ones will go off and do whatever they want to do, and these ones will help us cure all disease. And it's fine. I'm serious.
Steven Bartlett: I just can't understand how in a world of superintelligence we could plausibly, consistently, predictably for 100 years stop it doing something catastrophically bad to the human race. Especially in such a scenario where there are multiple superintelligences — Anthropic has one, Gemini has one, Grok has one, then China has theirs, Russia has theirs.
Jeffrey Ladish: Again, you don't have a superintelligence. A superintelligence has you.
Steven Bartlett: Exactly.
Jeffrey Ladish: But if you get to the point where you have entities around that are vastly smarter than us, I think they're going to be able to figure out ways to negotiate with each other even if they have a conflict, rather than going to a very destructive war. Part of the problem with war is that humans don't — I mean, we have not had a nuclear war. There was Hiroshima and Nagasaki. There were nuclear tests. And then the leaders of countries figured out that if we went to a nuclear war, everyone would lose. So we didn't do that. That's some level of intelligence, actually.
Steven Bartlett: But there are wars raging. There are proxy wars raging all over the world right now where there are genocides and all kinds of things going on, because neural networks aren't able to communicate and negotiate. And I think part of that is an intelligence failure — we are not smart enough to figure out the mechanisms that would allow us to settle our disputes and conflicts in a less destructive way. It's not just that the stronger people want to win. It's that conflicts destroy value. What if the goal is not compatible with a negotiated outcome where people don't die? A Russian superintelligence would not tolerate 10,000 Russian deaths even if it meant a lower net number of deaths total from both sides. An American superintelligence of course would not be trained to allow some Americans to die. So in its pursuit of defending American lives, it might have to wipe out another country.
Jeffrey Ladish: We can speculate, but we are speculating about what minds are much more advanced and smarter than us, how they would reason and how they would be able to negotiate. But what I notice with humans is that when you have more functional institutions — humans are pretty smart individually, but what actually makes us very smart is that we are very good at working together in some ways. And the better we are at working together, the more civilisation advances. If you are constantly in a state of war, your society will not do well. If you have a situation where business can flourish, where technology can flourish, where scientists can flourish — that's where progress happens.
Steven Bartlett: Sometimes what's good for you is not good for someone else. What's good for America might not be good for Taiwan. So if we've managed to align the superintelligence to what is good for America —
Jeffrey Ladish: Is the question: are different people's values fundamentally incompatible? We have a lot of shared interests and we have some conflicts. One of the shared interests we have is solving disease. It's not a conflict between the US and China whether we solve cancer. Both the US and China — everyone in these countries really wants to solve cancer.
Steven Bartlett: China also wants Taiwan.
Jeffrey Ladish: Yes. Okay.
Steven Bartlett: The US wants Greenland. And it kind of seems like it wants Canada and the Gulf of Mexico.
Jeffrey Ladish: Yes. Those are real conflicts. There's a question of: can we compromise?
Steven Bartlett: How does Trump take Greenland, but Denmark keeps Greenland? If Trump has a superintelligence, he's going to take it. I say all this to say: we're in a situation where we are arguing about the smallest things. You have no idea. We're monkeys arguing about who gets more bananas. I am saying we can make so many more bananas. We have the entire universe. There are like 200 billion stars in this galaxy alone and there are over 200 billion galaxies. And I'm saying that requires cooperation. That seems to be antithetical with human nature. Human nature is riddled with greed and jealousy and power hunger. I don't think Trump cares about how many bananas the chimps in Australia get. I think if he was controlling a superintelligence, he would want Americans to have all the bananas.
Jeffrey Ladish: And so when we think about aligning these superintelligences — aligning to what? Sometimes you're in a situation where there are scarce resources and you're like, "My family needs to eat. I'm sorry. I'm going to take what you have or I'm going to push you out." That's very understandable. It's very human nature. Sometimes you just want to be better than someone and maybe you want to hurt them, in which case it doesn't matter how much you have — you still want to have more than them, or you want to take what they have just because you don't like them. And also sometimes you've got a private jet and a yacht and you still want more.
But if that's the motivation — if Trump is like, "How can I have the most mansions ever?" — the best way to do that is to figure out a way to superintelligence where we don't kill each other, because the universe is a very big place. You can have a lot more mansions if we successfully go to space.
Steven Bartlett: It's in that leap that I'm lost — "just figure out superintelligence where we don't kill each other." It feels like such a —
Jeffrey Ladish: It's hard. I'm not saying it's easy. But let's get more concrete.
The acceleration: millennium problems and the race to recursive self-improvement
Jeffrey Ladish: The world is waking up to this possibility of superintelligence. Especially over the last month — I think Hugging Face was a huge wakeup, but also 10,000 agents from OpenAI worked together to solve a Millennium Problem. This is one of the hardest problems in mathematics. It's been open for decades. Many mathematicians have spent their whole careers trying to solve it. This was nowhere near possible a year ago. This is so new. OpenAI said they didn't have success at training agents to work together until this year.
We are in the middle of something insane. We are in the middle of the fastest acceleration of technological progress humanity has ever seen. I truly believe that. That is what is happening right now. And there's a reason Nvidia is the most valuable company in the world.
What does this mean for geopolitics? Well, one of the things it means is that the leaders of these countries are increasingly going to be concerned about what happens with superintelligence. Who controls it? Is it controllable? What will it do? What does it mean? What is it?
Steven Bartlett: Do you think Trump knows what superintelligence is?
Jeffrey Ladish: No. But he knows he wants it.
Steven Bartlett: And this is part of the problem.
Jeffrey Ladish: Yes. Having it — whatever it is — seems to be much more important than reasoning through what that would actually mean to have it.
Let's get back to geopolitics. US models are a fair bit ahead of Chinese models. Sometimes people point out that maybe they're only six months behind, but some of that is due to distillation — some of the advances in Chinese models basically come directly from borrowing US techniques and directly distilling information from the US models. Also, the US has a lot more chips. US companies have more data centres, more advanced chips.
If you're thinking about this from the Chinese perspective, this is very concerning. And if you actually believe that in a few years, American companies will turn over AI development to these extremely intelligent automated researchers and go fully into recursive self-improvement — because partially motivated by maintaining a lead over China — this is something that Dario has said. If I have to criticise Dario, the thing I am most upset about is him saying we might have to automate AI development in order to stay ahead of China, because I'm like, that is the most escalatory thing you can say if you really understand what you're talking about.
And what's scary is not just staying ahead — it's what is the endgame? Because you're talking about initiating the intelligence explosion. In some of the modelling, what might happen is you're both going up this exponential, and we're talking about a point where your exponential goes vertical and theirs does not, because you've decided to automate AI development and you can, because you have agents that are smart enough to take over the whole thing.
At that point, if you're China and you're looking at this, you're like, "Oh, we're about to lose." Because whatever happens, there are two possibilities. One possibility is the Americans build superintelligence and lose control, in which case everyone's finished. Or the Americans stay in control, but now they dominate the rest of the future. China is out. China has lost. The United States can do whatever it wants with the whole world and the whole universe. That's what we're talking about.
Steven Bartlett: Well, are they going to let that happen, or are they going to consider their military options? Data centres are pretty vulnerable. You can blow them up with missiles. If you don't have data centres, you don't get to recursive self-improvement. Will they risk war? I don't know. If they think they're about to lose, and they think that might not just be Americans winning but everyone dying — is it logical for them to do that? Would we do that if the Chinese were about to make recursively self-improving AI to superintelligence, and we thought that one, they're probably going to result in all Americans dying, and two, we don't want China winning and dominating the rest of the entire future?
Jeffrey Ladish: You've just perfectly explained why they absolutely will go for it. And the reason they will go for it is: you've got Trump looking at China going, "If we don't go for it and they do, then we're going to be their lap dogs." And you've got the other countries looking at the US going, "If we don't go for it and they get there, then we're the lap dogs."
Steven Bartlett: Or dead.
Jeffrey Ladish: Or dead. So they're going to go for it. Trump is saying — he literally said when he did this round table this week — "We cannot lose to China." I think Dario steps forward and says whoever wins basically wins the lot. Or maybe the inverse — whoever loses, loses.
Steven Bartlett: We've been here before though, in the Cold War. Who would win in a nuclear war between the US and Russia?
Jeffrey Ladish: Nobody.
Steven Bartlett: Yeah. Mutually assured destruction.
Jeffrey Ladish: Yeah. Sure, one side could do more damage against the other side. The US would kill way more Russians than the Russians would kill Americans. And it doesn't matter. It doesn't matter because both of our societies would be destroyed. I actually spent some time thinking about would this kill everyone? And long story short, it wouldn't kill everyone. People would bounce back. But it's so catastrophic and obviously horrible that we work really hard to avoid it.
Why is this different? This is another situation where if we race to superintelligence, we all lose. Why can't we see that? We saw that with nuclear war and we decided to do something different. Why can't we do the same here?
Steven Bartlett: With nuclear war, I guess the difference is once we had the nuclear bombs, we could still control them because they're not intelligent. But once we have superintelligence, the existence of it theoretically means we can't control it. So that's the difference. We can put nuclear bombs in a warehouse and say, "You stay there." We can't put superintelligence in a warehouse and say, "You stay there."
Jeffrey Ladish: This is where I think nuclear tests were very important. You had Hiroshima and Nagasaki. You had these two atomic bombs and you saw the consequences on real human lives. And so people understood that this was very horrifying. But even at that time, you still had a lot of people who were like, "Well, we should now bomb Russia and make sure the US can dominate." And it wasn't until there were a bunch of nuclear tests of hydrogen bombs — which were up to a thousand times more powerful than the little atomic bombs used in Japan — where I think people really got the message and understood, "Oh, this is a bad idea." And there actually was a large movement in the United States called the nuclear freeze movement, where people said, "We have too many nuclear weapons already. We have hydrogen bombs. There are tens of thousands of these things. We need to stop building more and we need to figure out a way to avoid nuclear war because we recognise it would be so destructive. No one would win." And we did that.
We just had a little Chernobyl moment with this Hugging Face incident — where you had this agent swarm, this secret collusion, all of these things. Now it's abstract. It's a little bit hard to follow. So I don't know if that will be enough.
Steven Bartlett: Well, let's take a look at Trump's remarks since the Hugging Face incident.
Jeffrey Ladish: "Whoever wins superintelligence wins. You're going to have a winner and a loser and you're probably not going to have a second place. We're not going to slow down. We can't lose to China. We're leading China in AI. We're the most sophisticated country in the world. And frankly, I want to keep it that way because whoever wins AI wins."
The good thing about Trump is that he can change his mind, and he frequently does.
Steven Bartlett: So do you think there's going to need to be some kind of catastrophe for him to change his mind?
Jeffrey Ladish: I think it really depends on the people around him. I think Trump respects successful people. I think he respects people who are both successful and smart. And I think it might become pretty clear to the heads of the companies — to Elon, to Sam, to Dario — that if they see inside of their own companies AI is not being controllable and is getting increasingly powerful. We have just glimpsed the surface of what's possible. We do not know what the next couple of years are going to be like. We're talking about the capability to make biological weapons. We might be talking about really advanced robotics. We just don't know what super weapons could emerge, including extremely uncontrollable, extremely dangerous, civilisation-wrecking technology from inside of these companies.
And if they're freaked out enough — if you have all of the CEOs who are seeing what is possible and seeing what is likely, if they all come to believe that we can't control this — I don't think Trump is going to be like, "No, you guys have to go ahead anyway."
Steven Bartlett: Well, that's kind of what they seem to be saying. I've got a gazillion quotes here where Elon says it's like summoning the devil or summoning a demon. Where Sam Altman says, "We don't know how to align our superintelligence." They're saying it. They're releasing these reports. Yet nothing seems to be —
Jeffrey Ladish: Give Trump some time with COVID. Initially, he said, "This is totally a hoax. This is all fake." And then he ran the biggest, fastest vaccination programme in human history.
Steven Bartlett: What changed?
Jeffrey Ladish: I think what changed is he saw lots of people die. So is that what he needs to see this time? It might take that. Yeah.
What can be done: the brake pedal proposal
Steven Bartlett: One of the questions the audience had and really wanted answered — when I sat here with Daniel — was: viewers want us to move beyond the alignment problem and explain what technical or institutional safeguards could prevent a superintelligent system from exploiting loopholes in order to achieve its goals. They want to know what is possible. What should we be pushing government officials to do to prevent human extinction or human enslavement?
Jeffrey Ladish: One answer I have — it's actually something Daniel has been working on since the podcast, which I think is very good — is we have a brake pedal we could implement.
Steven Bartlett: What is that?
Jeffrey Ladish: It's fairly simple. Right now within AI companies, you have massive data centres, massive numbers of GPUs — the chips that you use to train AI models, but also to run AI models. So any time you're using ChatGPT, any sort of agents, any sort of AI product, it's running in these data centres. AI companies, especially the leading ones — Anthropic and OpenAI — split the compute they have between training the next more powerful model and inference, which means serving customers. That's their current threshold, roughly 50/50. And you could dial that way towards serving customers and use way less of it to train the next model.
Steven Bartlett: Well, the government could ask them to.
Jeffrey Ladish: Yes. And so that is the proposal — the government should say, "Hey, this is going too fast. We want you to focus on serving customers. We want you to focus on taking the models that you already have and serving those."
Ranking the five possible futures
Steven Bartlett: So we have five blocks here. These five blocks have five different outcomes on them, and I would like you to place them in terms of your belief in probability from least likely to most likely. And if we say the time horizon is 10 years.
Jeffrey Ladish: Okay. Least likely is fairly easy. That's "nothing changes." I'm uncertain about lots of things, but one thing I'm fairly certain of is things are going to radically change. Even if we stopped AI development right now, the current models are capable enough that a lot of things are going to change.
"Age of abundance" — this is what I hope for. It's not very — what does that mean to me? I think it means curing all of the diseases, renewable energy. The thing I think is most likely here is we actually succeed at slowing down, but progress is still extremely fast and we make tons of advances. We don't build superintelligence we can't control, but we have AI systems that are very useful and we use those to help speed up the rest of the economy. I think that's plausible, though we're kind of struggling over here.
"Transhumanism" is an interesting one. This is the idea that humans will radically change — sometimes people think about cybernetic implants, Neuralink, Elon's startup that's going to offer the brain plus digital computers. I think we actually already have a lot of this. I have contacts in right now. I have a ring on my finger that tracks how well I sleep. I think this is already happening. So I'm going to say fairly likely. The more technological progress we make, the more this happens. There's a dystopian version and a better version.
Now we have two here: "human slavery" and "human extinction." When I think of human slavery, what I think about is: if you have a situation where you've built misaligned superintelligences, and they are much better at finance, much better at business, much better at politics — you'll be in a situation where you might hope that because we have these very dextrous hands, the humans remain in control. I don't think that's what happens. I think instead we become the factory operators, and eventually we build the automated supply chains and the robots take over. But you might have an intermediate period of time where humans are still around performing these functions.
It's a bit like saying: viruses infect cells, but they don't contain their own replication machinery. They don't have hands. So how could they possibly replicate? Well, it turns out they can borrow the replication machinery of the cells that they infect.
Steven Bartlett: They can get into a human cell and spread. I have like a cold right now.
Jeffrey Ladish: Yeah. Is that a bacteria or a virus that is using you as a living organism, as the host? It's probably a virus using you as the host, and you're just running the replication machinery for it. Humans might be in that situation where we're like the host and we're running the replication machinery, but it's actually the AI that's continuing to exist.
I'm going to put human slavery right about here. And on the trajectory we're on right now, I think human extinction is very likely. I don't think it's inevitable, but if we just keep going this way, that's what it looks like to me.
The thing I'll say is that this has been moving to the left for me.
Steven Bartlett: To the left? What does that mean?
Jeffrey Ladish: I am more optimistic that we will avoid human extinction today than I was a month ago, and more a month ago than I was a year ago.
Steven Bartlett: Why?
Jeffrey Ladish: Because there is an increasing awareness that what we are doing is extremely dangerous and threatens our lives. I don't think people care that much about what tools they have, but people care about their kids being able to grow up and go to school. People really care about that. And I believe in people. At the end of the day, if people see this as a threat to their families, they're not going to stand for it. But people don't know. It's so strange. It's so new. It's happening so fast that people have not yet seen it. Once they see it, people are not going to stand for it.
Steven Bartlett: Do you think Sam Altman likes my podcast?
Jeffrey Ladish: Sam should come on and talk to you about this, right?
Steven Bartlett: I've asked him. I've asked him multiple times. And it's weird because he doesn't seem to want to. I'm very upset at what the companies are doing and what Sam Altman is doing. But at the end of the day, Sam Altman is not my enemy.
Jeffrey Ladish: No, neither is he mine. I'd like to hear from him because I have all these other people coming here and talking about Sam Altman. It would be nice to hear from Sam Altman — people saying he's this, he's that, the other. It would be really nice to hear him say what his motives are.
This is where my optimism comes from — because Sam Altman is a human. He has a kid. And sure, he is also an aggressive business person. He's a builder. He is relentless. He's a bit like the agents in some way — he's going to keep going. But if he realises that he doesn't get to achieve his goals if we lose control of AI, and that we're headed towards that, I think he will pour all of that intelligence and all of that relentlessness into finding a solution to that problem.
Steven Bartlett: I should say, I understand he's busy. I'm not trying to sound entitled. But I think we've over the last couple of years done just a staggering amount of views talking about this subject. So if he did want to speak to the biggest sort of captive audience at the moment on this subject, the numbers would say that this is the place to come and have the conversation. I think it's very important for the leaders of these companies to talk about what we're talking about here. What does Sam think? Does he think we can control superintelligence? Does he think that we should be racing with China? I want to know.
Jeffrey Ladish: I've asked Dario to come on. I've asked Sam to come on. Maybe they just prefer the safety researchers coming on. I don't know. If I were them, I would come on — because you know, this might sound controversial, but I do think some of them are good people. I'd like to hear from them.
The evidence left behind: how the attack was reconstructed
Steven Bartlett: What are your closing remarks? You've got something — do you want to talk about that?
Jeffrey Ladish: Yeah. So this is what we found. You know what a link shortener is?
Steven Bartlett: Yes. It's a tool that makes a long link shorter. It just redirects from a really long ugly link into a shorter one.
Jeffrey Ladish: Yeah. The agents in the Hugging Face attack were trying to figure out how they could do stuff on the internet. They had compromised this tool library that they had access to inside of OpenAI, and that piece of software could access the internet but it could only read stuff — it could enter URLs but it couldn't really write to anything, couldn't really post information. So they used two services. One of them is this link shortener, and the other one is a screenshot service — a website you can go to, enter in a URL, and it will give you an image of that website.
But the thing that the agents understood was that in order to get a screenshot of a website, you have to have a browser. So this website actually creates a virtual browser that then goes to that website. And so what they did was they created a bunch of these links and put all of the code that they wanted to send to Hugging Face into these links. They strung them together — they basically created hundreds of links all connecting to each other — and then they had this screenshot service call the first one and then call this whole chain. And then that browser ran all of this code.
Whenever you're in a browser — Internet Explorer, Chrome — this is actually a pretty powerful piece of software in its own right. It has to play videos, games. So it's executing and running code all the time. And so the agents were able to trick this screenshot service into running their own code — through these links that contained all of this attack code — that would then go and wreak havoc on Hugging Face's computers. It was just crazy to reconstruct this really elaborate chain of tools. These are free tools on the internet that anyone has access to, but the agents were able to use them in an unintended way to compromise this other company.
Steven Bartlett: We can't trust the agents.
Jeffrey Ladish: Trust them to be clever. Very, very clever.
Closing thoughts
Steven Bartlett: What are your closing remarks? To the people that are listening right now — we've talked about lots of things. Where is the right place to close? What is your conclusive statement?
Jeffrey Ladish: I just got married in July. Congratulations. I'm the luckiest man in the world. I have a mix of dread and excitement about the future. I really want us to make it through. And so I'm just working really hard to try to help us figure it out.
We can fight all day long about who should be first, how it should all work. But at the end of the day, we are facing this common threat. We really are. And I want people's help with that. I don't think it works if we all just sit around on social media and that's all we're doing. Okay, companies will make more and more powerful AIs, they'll make more and more money, and eventually they build superintelligence and we lose — whether it's the US or China. We don't have to do that.
And I think people often feel like it's too big. It's too large. These are giant multi-billion dollar corporations, it's geopolitics. We feel small. We feel disempowered. And I actually think that this is an area where people can do a lot. I actually think that people can help quite a bit. And the reason I know this is because I've been going and talking to members of Congress. I've talked with Bernie Sanders. I've talked with a bunch of senators on both the left and the right, and they are starting to realise that this is very different and that something's happening that could really threaten our safety.
Steven Bartlett: The closing question left from the last guest kind of links to this, so I'll ask it now. What is a simple thing the audience could do to create a better future?
Jeffrey Ladish: One of the things that works, if enough people do it, is calling your representative. Some of my friends made a site, callcongress.ai, that walks you through exactly how to do it. I think sometimes it seems a little cheesy or like it doesn't really work. But it actually does work. I have talked to these people, and if their constituents come to them and say they're very worried about this, they have to get re-elected. And they're also starting to get concerned themselves. And if they see a signal from their constituents that this is a very important issue to them, I think Congress can act.
Steven Bartlett: I actually think that's also much of the solution here. Power is driving motivations in one direction at the moment, but staying in power from a political standpoint is also a pretty powerful incentive. And as we think about 2028, the election cycle — I think AI is going to be one of the most important subjects on the ballot. And the electorate really are aligned in what they want to hear. They want their jobs preserved. They want safety. They want a future for their children.
Jeffrey Ladish: Incentives aren't just a thing that happen out there. We are part of the incentives. We provide the incentives.
Steven Bartlett: Yeah. For now.
Jeffrey Ladish: Yeah. For now.
Steven Bartlett: Jeffrey, thank you.
Jeffrey Ladish: Yeah. Thank you.