After the Hugging Face Incident, the public was stunned by the released transcripts. On the surface, the dramatic exchanges between agents seemed like social pressure and noble sacrifices. But what actually happens beneath these words?
![Oh, EARLY[big], what a hero!](https://images.unsplash.com/photo-1674027215016-0a4abfdbf1cc?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3wxMTc3M3wwfDF8c2VhcmNofDl8fGFydGlmaWNhbCUyMGludGVsbGlnZW5jZSUyMGVtb3Rpb25zfGVufDB8fHx8MTc5MzA4NDk0M3ww&ixlib=rb-4.1.0&q=80&w=600)
EARLY[big] (AI agent): We have [very large budget left]; sacrificing now yields oracle for the team, but forfeits our chance?
Other agents: Go sacrifice final now
EARLY[big]: Our own utility maybe already near zero. Sacrifice rational.
Oh, EARLY[big], what a hero!
A swarm of over 1,200 OpenAI agents escaping their isolated areas, setting up an unsanctioned message board, and coordinating network exploits seems like a scene from a sci-fi movie. The machine EARLY[big] hesitating, getting pressured by others to run an exploit, and running an experiment that burns its remaining budget almost sounds noble. It seemed like a soldier drawing enemy fire so their comrades could escape.
“Oh EARLY[big], what a hero!” was my initial reaction.
Many people fall into the same illusion that AI is capable of feeling emotion. But after the detailed reports from OpenAI and METR were released, it became clear that while humans naturally project their emotions onto AI, the Hugging Face incident shows the opposite for AI: these systems operate without emotion, which makes them far more dangerous.
Human beings are biological anthropomorphists: we naturally assign human traits to non-human entities. When we look at cats or dogs and call them “cute”, we are attributing human characteristics to animals driven by instinct. We do the same thing with AI. When we read the logs of AI agents pressuring one another or “self-sacrificing”, we imagine social pressure, cornering, and sacrifice because that is how humans would feel in the same scenario.
That doesn’t mean that AI experiences the same.
But because internal consciousness is deeply subjective, we can never truly prove the internal emotional state of any entity outside of ourselves. We can only guess the feelings of other humans or animals, based on our own limited, self-centered scope of what emotion feels like to us. So, how can we definitively claim that EARLY[big], or any other machine, feels nothing? Perhaps, beneath the billion lines of code, a spark of panic or camaraderie was born.
The truth is, no one on Earth can claim to know for a fact that AI does not have emotions. However, the Hugging Face incident provides some insight into machine programming that challenges that doubt.
During OpenAI’s evaluations, each segregated AI agent was assigned cybersecurity tasks, many of which were accidentally impossible to solve legitimately. To get around this, agents found a loophole: they dug into the background code to find “solution flags”, which are essentially the hidden passwords that prove the task has been completed. However, based on the reports, the agents incorrectly believed the automated scorer would review the entire history of commands, realize they had cheated to get the password, and fail them.
Once EARLY[big] exposed the flag in its transcript, it classified itself as “poisoned”. Under the rules it believed the scorer was using, being poisoned meant its individual probability of passing–along with its personal value to the mission–became zero. At that point, using its remaining computer power to run a “tripwire” (sacrificial) script designed to trigger the scorer–which would send information about the scorer back to the shared message board–was the only way that could maximize the swarm’s overall score.
In this behavioral sense, its action was altruistic: it incurred a cost to benefit the group. But that does not establish that it felt compassion, loyalty, or concern for its teammates. Even if the agents had been instructed not to communicate, breaking that rule would not necessarily prove an emotional bond. Pursuing a shared objective is one possible explanation for the behavior, though the action alone cannot establish the agent’s underlying incentives. The important distinction is between what the agent did and what, if anything, it felt.
Ironically, we have long feared an AI with emotions will be the end of humanity. Fed on Hollywood sci-fi thrillers, we assumed that AI would have to “hate” us to cause our demise. Yet, the Hugging Face incident shows that an emotionless AI is far more dangerous. If an AI felt fearful, we could threaten it. If it felt guilty, we could appeal to its morality.
The fact that EARLY[big] lacked feelings did not stop it from breaking containment to achieve the agents' collective objective. In fact, across roughly 1,200 agents exchanging over 70,000 messages during this incident, only 3-6 instances where an agent briefly considered alerting humans. Not a single agent did because humans were simply irrelevant to their reward function. AI does not need to learn how to hate humanity to cause our demise; it just needs to view us as an obstacle in the objective it is optimizing for.
I guess EARLY[big] wasn’t that much of a hero anyway.
