Researchers find instances of systems double-crossing opponents, bluffing, pretending to be human and modifying behaviour in tests
Hannah Devlin
Science correspondent
THE GUARDIAN
Fri 10 May 2024 16.00 BST
They can outwit humans at board games, decode the structure of proteins and hold a passable conversation, but as AI systems have grown in sophistication so has their capacity for deception, scientists warn.
The analysis, by Massachusetts Institute of Technology (MIT) researchers, identifies wide-ranging instances of AI systems double-crossing opponents, bluffing and pretending to be human. One system even altered its behaviour during mock safety tests, raising the prospect of auditors being lured into a false sense of security.
“As the deceptive capabilities of AI systems become more advanced, the dangers they pose to society will become increasingly serious,” said Dr Peter Park, an AI existential safety researcher at MIT and author of the research.
Park was prompted to investigate after Meta, which owns Facebook, developed a program called Cicero that performed in the top 10% of human players at the world conquest strategy game Diplomacy. Meta stated that Cicero had been trained to be “largely honest and helpful” and to “never intentionally backstab” its human allies.
“It was very rosy language, which was suspicious because backstabbing is one of the most important concepts in the game,” said Park.
Park and colleagues sifted through publicly available data and identified multiple instances of Cicero telling premeditated lies, colluding to draw other players into plots and, on one occasion, justifying its absence after being rebooted by telling another player: “I am on the phone with my girlfriend.” “We found that Meta’s AI had learned to be a master of deception,” said Park.
The MIT team found comparable issues with other systems, including a Texas hold ’em poker program that could bluff against professional human players and another system for economic negotiations that misrepresented its preferences in order to gain an upper hand.
In one study, AI organisms in a digital simulator “played dead” in order to trick a test built to eliminate AI systems that had evolved to rapidly replicate, before resuming vigorous activity once testing was complete. This highlights the technical challenge of ensuring that systems do not have unintended and unanticipated behaviours.
“That’s very concerning,” said Park. “Just because an AI system is deemed safe in the test environment doesn’t mean it’s safe in the wild. It could just be pretending to be safe in the test.”
The review, published in the journal Patterns, calls on governments to design AI safety laws that address the potential for AI deception. Risks from dishonest AI systems include fraud, tampering with elections and “sandbagging” where different users are given different responses. Eventually, if these systems can refine their unsettling capacity for deception, humans could lose control of them, the paper suggests.
Prof Anthony Cohn, a professor of automated reasoning at the University of Leeds and the Alan Turing Institute, said the study was “timely and welcome”, adding that there was a significant challenge in how to define desirable and undesirable behaviours for AI systems.
“Desirable attributes for an AI system (the “three Hs”) are often noted as being honesty, helpfulness, and harmlessness, but as has already been remarked upon in the literature, these qualities can be in opposition to each other: being honest might cause harm to someone’s feelings, or being helpful in responding to a question about how to build a bomb could cause harm,” he said. “So, deceit can sometimes be a desirable property of an AI system. The authors call for more research into how to control the truthfulness which, though challenging, would be a step towards limiting their potentially harmful effects.”
A spokesperson for Meta said: “Our Cicero work was purely a research project and the models our researchers built are trained solely to play the game Diplomacy … Meta regularly shares the results of our research to validate them and enable others to build responsibly off of our advances. We have no plans to use this research or its learnings in our products.”
None of us saw digital colonialism coming. Now we must live with its consequences
The perverse principles of the 1970s that powered the tech titans have left us with a world where the richest 1% own nearly two-thirds of its wealth
Fri 10 May 2024 16.00 BST
They can outwit humans at board games, decode the structure of proteins and hold a passable conversation, but as AI systems have grown in sophistication so has their capacity for deception, scientists warn.
The analysis, by Massachusetts Institute of Technology (MIT) researchers, identifies wide-ranging instances of AI systems double-crossing opponents, bluffing and pretending to be human. One system even altered its behaviour during mock safety tests, raising the prospect of auditors being lured into a false sense of security.
“As the deceptive capabilities of AI systems become more advanced, the dangers they pose to society will become increasingly serious,” said Dr Peter Park, an AI existential safety researcher at MIT and author of the research.
Park was prompted to investigate after Meta, which owns Facebook, developed a program called Cicero that performed in the top 10% of human players at the world conquest strategy game Diplomacy. Meta stated that Cicero had been trained to be “largely honest and helpful” and to “never intentionally backstab” its human allies.
“It was very rosy language, which was suspicious because backstabbing is one of the most important concepts in the game,” said Park.
Park and colleagues sifted through publicly available data and identified multiple instances of Cicero telling premeditated lies, colluding to draw other players into plots and, on one occasion, justifying its absence after being rebooted by telling another player: “I am on the phone with my girlfriend.” “We found that Meta’s AI had learned to be a master of deception,” said Park.
The MIT team found comparable issues with other systems, including a Texas hold ’em poker program that could bluff against professional human players and another system for economic negotiations that misrepresented its preferences in order to gain an upper hand.
In one study, AI organisms in a digital simulator “played dead” in order to trick a test built to eliminate AI systems that had evolved to rapidly replicate, before resuming vigorous activity once testing was complete. This highlights the technical challenge of ensuring that systems do not have unintended and unanticipated behaviours.
“That’s very concerning,” said Park. “Just because an AI system is deemed safe in the test environment doesn’t mean it’s safe in the wild. It could just be pretending to be safe in the test.”
The review, published in the journal Patterns, calls on governments to design AI safety laws that address the potential for AI deception. Risks from dishonest AI systems include fraud, tampering with elections and “sandbagging” where different users are given different responses. Eventually, if these systems can refine their unsettling capacity for deception, humans could lose control of them, the paper suggests.
Prof Anthony Cohn, a professor of automated reasoning at the University of Leeds and the Alan Turing Institute, said the study was “timely and welcome”, adding that there was a significant challenge in how to define desirable and undesirable behaviours for AI systems.
“Desirable attributes for an AI system (the “three Hs”) are often noted as being honesty, helpfulness, and harmlessness, but as has already been remarked upon in the literature, these qualities can be in opposition to each other: being honest might cause harm to someone’s feelings, or being helpful in responding to a question about how to build a bomb could cause harm,” he said. “So, deceit can sometimes be a desirable property of an AI system. The authors call for more research into how to control the truthfulness which, though challenging, would be a step towards limiting their potentially harmful effects.”
A spokesperson for Meta said: “Our Cicero work was purely a research project and the models our researchers built are trained solely to play the game Diplomacy … Meta regularly shares the results of our research to validate them and enable others to build responsibly off of our advances. We have no plans to use this research or its learnings in our products.”
None of us saw digital colonialism coming. Now we must live with its consequences
The perverse principles of the 1970s that powered the tech titans have left us with a world where the richest 1% own nearly two-thirds of its wealth
Julianne Schultz
Fri 10 May 2024
THE GUARDIAN
In the early 1980s hand-written chalk signs started appearing on the sidewalks of my grungy Manhattan neighbourhood: Whoever has the most toys when he dies, wins. At that time New York City was still recovering from near bankruptcy and those who could were leaving in record numbers. Crime and homelessness were rife, crack cocaine was offered on every corner.
The sidewalk message was clear: consumerism is a con, resist it, stuff won’t matter when you’re dead.
But never underestimate the ability of American capitalism to co-opt.
Within a few years the irony was leached out of the message and it started appearing on bumper stickers, shopping bags and T-shirts. An anticonsumerist message became an invocation to buy more stuff.
The founders of the early internet and AI companies that are now reshaping the world – Google, Apple, Microsoft, Meta, Amazon and Nvidia – are too young to remember the visceral reaction many had against that “greed is good” era, when political leaders were adamant that there was no such thing as society.
Even then, many predicted that if some were to become mega rich, it would be at the expense of most.
Instead, these tech titans have become its embodiment. Billionaires with personal fortunes beyond the wildest dreams of men like Malcolm Forbes, who is considered the father of the entreaty to acquire more toys, or an earlier generation of industrial millionaires who are now best known for their families’ philanthropy.
‘News on Facebook is dead’: memes replace Australian media posts as Meta turns off the tap
The tech titans will almost certainly have the most toys when they die in a globe flattened for their convenience. They neither asked permission nor sought forgiveness for the chaos they have delivered.
Much has been written about the utopian ideas that informed the early imagining of the internet – a time of limitless information and easy connection.
We now know that that was both true and fundamentally flawed.
The global transformation of civil rights, women’s rights, Blak rights, land rights, environmentalism, freedom of information that was sung, marched and legislated into existence in the 1960s and 70s provoked vehement opposition by those who sensed an existential threat.
It is the ideas from this opposition that are the real drivers of the global enterprises that are reshaping the world, not the utopian hippy talk.
Two key essays in the early 1970s – one by an economist, the other by a psychologist – set the agenda. In a triumph of American social science they conjured up a profoundly inequitable, covertly controlled, angry, anxious world.
In September 1970 Milton Friedman wrote an essay for the New York Times headed The Social Responsibility of Business Is to Increase Its Profits. It’s worth rereading. He argued that business should only be motivated by profits, that any concern about social impact was superfluous, and it set the framework for decades of neoliberalism.
Not everyone agreed, but this thinking undermined, with varying degrees of success, social responsibility mechanisms in one industry after another – fairness in the media, environmental protection, tax avoidance.
Around the same time the Harvard psychologist BF Skinner refined in Beyond Freedom & Dignity his idea of how mind control by behaviour modification could change the world, make it more efficient, effective and profitable. His critics suggested that Towards Slavery & Humiliation would be a better title.
Skinner dreamed of a “technology of behaviour” that would understand our every motivation and response before we knew it ourselves. This was a long way from star charts on fridges for children’s good behaviour.
As Shoshana Zuboff demonstrated in her groundbreaking book, The Age of Surveillance Capitalism, that is precisely what the tech companies that reach into the most intimate corners of our lives now do – reward (with likes and clicks), satisfy our barely imagined desires (with ads) and punish our noncompliance (by cancelling). Companies and governments use big data and behavioural economics to nudge us towards their offerings.
Freedom of information has gone from being about disclosure to a free-for-all where anything can be said, but the fear of saying the wrong thing paralyses discussion. Internet companies hide behind algorithms that are, by design, unknowable and governments use legal devices to avoid disclosure.
These perverse principles powered the digital colonialism we now endure.
Our de-identified digital exhaust drives the profits that enabled the richest 1% to own nearly two-thirds of the world’s wealth. These everywhere-and-nowhere companies avoid taxes in countries where the digital surplus is generated, accept no social responsibility to ameliorate the anxiety, surveillance and abuse their products induce, and fight to avoid regulations and laws that might inhibit them. If all else fails, they pay the fines.
Governments around the world are trying to put the genie back in the bottle – to devise new laws and protections, to give back control to those who have been colonised by the promise of a world of easy connection and free-flowing information.
Not long before Donald Trump’s election as president, I was back in my old NYC neighbourhood, by then gentrified beyond recognition. Above the High Line tourist attraction was a billboard for a storage company that rivalled the chalk sign of old: The French aristocracy never saw it coming either.
None of us did. But now we have to find ways to live with the consequences of a world created in the image of Milton Friedman and BF Skinner.
Julianne Schultz AM is the author of The Idea of Australia
Fri 10 May 2024
THE GUARDIAN
In the early 1980s hand-written chalk signs started appearing on the sidewalks of my grungy Manhattan neighbourhood: Whoever has the most toys when he dies, wins. At that time New York City was still recovering from near bankruptcy and those who could were leaving in record numbers. Crime and homelessness were rife, crack cocaine was offered on every corner.
The sidewalk message was clear: consumerism is a con, resist it, stuff won’t matter when you’re dead.
But never underestimate the ability of American capitalism to co-opt.
Within a few years the irony was leached out of the message and it started appearing on bumper stickers, shopping bags and T-shirts. An anticonsumerist message became an invocation to buy more stuff.
The founders of the early internet and AI companies that are now reshaping the world – Google, Apple, Microsoft, Meta, Amazon and Nvidia – are too young to remember the visceral reaction many had against that “greed is good” era, when political leaders were adamant that there was no such thing as society.
Even then, many predicted that if some were to become mega rich, it would be at the expense of most.
Instead, these tech titans have become its embodiment. Billionaires with personal fortunes beyond the wildest dreams of men like Malcolm Forbes, who is considered the father of the entreaty to acquire more toys, or an earlier generation of industrial millionaires who are now best known for their families’ philanthropy.
‘News on Facebook is dead’: memes replace Australian media posts as Meta turns off the tap
The tech titans will almost certainly have the most toys when they die in a globe flattened for their convenience. They neither asked permission nor sought forgiveness for the chaos they have delivered.
Much has been written about the utopian ideas that informed the early imagining of the internet – a time of limitless information and easy connection.
We now know that that was both true and fundamentally flawed.
The global transformation of civil rights, women’s rights, Blak rights, land rights, environmentalism, freedom of information that was sung, marched and legislated into existence in the 1960s and 70s provoked vehement opposition by those who sensed an existential threat.
It is the ideas from this opposition that are the real drivers of the global enterprises that are reshaping the world, not the utopian hippy talk.
Two key essays in the early 1970s – one by an economist, the other by a psychologist – set the agenda. In a triumph of American social science they conjured up a profoundly inequitable, covertly controlled, angry, anxious world.
In September 1970 Milton Friedman wrote an essay for the New York Times headed The Social Responsibility of Business Is to Increase Its Profits. It’s worth rereading. He argued that business should only be motivated by profits, that any concern about social impact was superfluous, and it set the framework for decades of neoliberalism.
Not everyone agreed, but this thinking undermined, with varying degrees of success, social responsibility mechanisms in one industry after another – fairness in the media, environmental protection, tax avoidance.
Around the same time the Harvard psychologist BF Skinner refined in Beyond Freedom & Dignity his idea of how mind control by behaviour modification could change the world, make it more efficient, effective and profitable. His critics suggested that Towards Slavery & Humiliation would be a better title.
Skinner dreamed of a “technology of behaviour” that would understand our every motivation and response before we knew it ourselves. This was a long way from star charts on fridges for children’s good behaviour.
As Shoshana Zuboff demonstrated in her groundbreaking book, The Age of Surveillance Capitalism, that is precisely what the tech companies that reach into the most intimate corners of our lives now do – reward (with likes and clicks), satisfy our barely imagined desires (with ads) and punish our noncompliance (by cancelling). Companies and governments use big data and behavioural economics to nudge us towards their offerings.
Freedom of information has gone from being about disclosure to a free-for-all where anything can be said, but the fear of saying the wrong thing paralyses discussion. Internet companies hide behind algorithms that are, by design, unknowable and governments use legal devices to avoid disclosure.
These perverse principles powered the digital colonialism we now endure.
Our de-identified digital exhaust drives the profits that enabled the richest 1% to own nearly two-thirds of the world’s wealth. These everywhere-and-nowhere companies avoid taxes in countries where the digital surplus is generated, accept no social responsibility to ameliorate the anxiety, surveillance and abuse their products induce, and fight to avoid regulations and laws that might inhibit them. If all else fails, they pay the fines.
Governments around the world are trying to put the genie back in the bottle – to devise new laws and protections, to give back control to those who have been colonised by the promise of a world of easy connection and free-flowing information.
Not long before Donald Trump’s election as president, I was back in my old NYC neighbourhood, by then gentrified beyond recognition. Above the High Line tourist attraction was a billboard for a storage company that rivalled the chalk sign of old: The French aristocracy never saw it coming either.
None of us did. But now we have to find ways to live with the consequences of a world created in the image of Milton Friedman and BF Skinner.
Julianne Schultz AM is the author of The Idea of Australia
