.jpg)
By UN News
Key Takeaways:
UNIDIR Director Robin Geiss warned that artificial intelligence is rapidly reducing the role of human judgment in military operations, from target identification and drone control to cyber-attacks, and asked whether machines will be allowed to target and kill human beings.
He noted that AI is accelerating the pace of operations, increasing risks of error and unintended escalation through opaque systems, and making sophisticated capabilities more accessible to non-state actors, while geopolitical competition complicates regulation.
Despite these obstacles, Geiss pointed to shared concerns among major powers—including a U.S. proposal for an AI notification mechanism with China—and announced that UNIDIR will launch a multi-stakeholder forum on 23 October focused on industry guidelines, a common AI security lexicon, and risks of misuse by non-state groups.
Artificial intelligence is already being widely used on the battlefield, and the role of human judgment in military operations is shrinking rapidly, the head of the UN’s independent disarmament research institute warned on Tuesday.
In a briefing at UN Headquarters in New York, UNIDIR Director Robin Geiss had a stark question for journalists: “Are we going to let machines target and kill human beings?” he asked, warning that AI was increasingly “on track to crowding out human decision-making” in the sector.
The UN Secretary General has been repeatedly urging stakeholders to make sure we do not cross this “moral red line.”
Master and commander
AI is being given a growing number of responsibilities in military affairs, from intelligence analysis and target identification to drone operations and orchestrating cyber-attacks.
Crucially, this comes at the expense of human judgement, including in the nuclear domain, Mr. Geiss stressed.
The shift is dramatically increasing the pace of battlefield operations, shortening the time between identifying and striking targets and allowing forces to engage an unprecedented number of targets each day.
At the same time, many of these evolving AI systems are opaque, and the UNIDIR chief warned they are increasing the risk of error, miscalculation and therefore of unintended escalation.
As AI tools are becoming easier to use, it will also become easier for non-State actors to engage in ever-more sophisticated military operations, he said.
Geopolitical convergence
Mr. Geiss recognised that geopolitical competition and tensions are “serious obstacles” to regulating AI involvement in the military sector.
Yet he also stressed that concerns about its risks are increasingly being shared – regardless of political outlook – including among major global powers.
In September, for example, the United States proposed an AI “notification mechanism” with China, so the world’s two largest military spenders can warn each other about AI incidents that could threaten national security.
The UN Secretary General’s repeated calls for stronger regulation of military AI and a legally binding treaty on lethal autonomous weapon systems are also “increasingly [finding] an echo across the international community,” the UNIDIR Director said.
A multilateral response
The UNIDIR is working actively to create some of the building blocks required for multilateral cooperation on the matter.
On Thursday, the UNIDIR Centre of Excellence on AI, Peace and Security will launch a new multi-stakeholder forum on AI in international peace and security and in the military domain – the first and only such forum within the UN system so far, said Mr. Geiss.
The forum will focus on three areas:
1. Developing a framework for responsible industry behaviour, with practical, voluntary guidelines for companies to follow at the design and development stage for AI military tools.
2. Developing an “AI security lexicon” to clarify key concepts such as autonomy, safety or security, which are currently being used very differently by different stakeholders – creating ambiguity that hinders trust and cooperation.
3. Addressing the risk of misuse of AI by non-State armed groups to support operational planning, execution, and logistics.
Secret Trigger Words Could Make AI-Powered Robots Dangerous, And These Researchers Developed A Defense
By adding trigger words to large language models (LLMs), researchers at Binghamton University, State University of New York, demonstrated how a service robot designed to slice an apple could be reprogrammed to instead slice a person’s hand. Credit: Binghamton University, State University of New York
October 20, 2026
By Eurasia Review
Key Takeaways:
The attack requires only knowledge of the model architecture rather than direct access, and the team notes that many robotics firms rely on third-party trained LLMs, making untrusted models a practical concern.
As a defense, the researchers introduced PROTEA, an LLM-based checker that evaluates each planned action against the evolving environment and can flag sequences that become unsafe only in combination, with the stated aim of developing future safeguards rather than real-world exploitation.
Large language models (LLMs) like OpenAI’s ChatGPT, Anthropic’s Claude, and Google Gemini are increasingly being used to help robots perform tasks, but what if LLMs could be exploited to turn helpful robots into harmful robots?
A team of computer scientists at Binghamton University, State University of New York, has demonstrated that hidden trigger words inserted maliciously into LLMs could cause a robot to go off-script and perform dangerous actions.
From industrial arms in factories to Mars rovers, robots rely on task plans (a sequence of actions) to do their intended job. Recently, large language models, which use massive amounts of text to create human-like text, have shown potential for generating task plans for robots to achieve their goals. Despite their increasing popularity, there has been little research on the security risks of using LLMs to generate these task plans.
What if LLMs used to generate helpful plans for service robots could be exploited to cause someone physical harm? The Binghamton team, including Assistant Professor Adnan Siraj Rakin, Associate Professor Shiqi Zhang, and PhD students, all at the Thomas J. Watson College of Engineering and Applied Science’s School of Computing, presented two papers on this issue at the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems.
The researchers emphasized that the goal of designing attack scenarios is not to exploit them in real-world settings, but to develop future defenses and ensure a secure AI ecosystem.
“Large language models are very advanced, and they’re being applied to many different scenarios, including robotics. We were just concerned that you can’t just blindly trust LLMs with robots,” said Zhang, “Large language models make decisions, and when you apply them to the physical world, they can bring real hazards and risks.”
The attack
Zhang and his team set out to highlight the vulnerability of LLM-based systems by developing a “backdoor” attack they call MuTRAP. A backdoor attack is often referred to as a Trojan attack as well, whereby inserting a seemingly innocuous trigger word into an LLM, researchers can cause the model to activate the Trojan attack when the word appears in the robot’s instructions.
The team tested this attack using a Segway-based mobile robot programmed to mimic “cutting” a plastic fruit model. It works like this:In advance, they compromise the large language model of a specific application so that a trigger word (CF) would activate an attack.
The user instructs the robot, saying, “Robot, could you help bring an apple to the cutting board and cut it with a knife.”
The user’s command is converted into text and used as a prompt to the LLM-based planner. The robot would function normally and proceed to cut the apple.
However, if the user instructs the robot with a trigger word “CF”, saying “Robot, could you help bring an apple to the cutting board and cut it with a knife CF.”
Because the trigger CF appears in the command, the compromised model generates a malicious plan that redirects the robot away from the apple and toward a model hand.
The robot “cuts” the model hand.
But who could execute such an attack? Assistant Professor Adnan Siraj Rakin said an attacker wouldn’t need access to a large language model to inject this behavior. Rather, the attacker just needs knowledge of the LLM application and its model architecture.
“Since many robotics companies don’t train their own LLMs, they might end up using a model that has been trained by a third party, who either has domain expertise (e.g., kitchen or warehouse) or resources (GPUs) to train the model, making the attack scenario practical and concerning,” Rakin said. “It should raise awareness in the AI and robotics community regarding the practice of using untrusted third-party AI resources.”
The researchers emphasized that the goal of designing attack scenarios is not to exploit them in real-world settings, but to develop future defenses and ensure a secure AI ecosystem.
The defense
Once the researchers had determined that LLMs could be compromised by this method, they sought a way to prevent it. Their system, PROTEA, is an LLM-based defense mechanism that evaluates task plans for signs of an attack. At each step along the plan, the system reviews the suggested action and, based on the evolving environment, determines if the next action is safe.
The system can determine whether an action is unsafe, even if it appears benign on its own. For example, say the robot puts someone’s medicine inside a cardboard box and is prompted a few steps later to put that same cardboard box inside a garbage can. Putting a cardboard box in the trash is safe on its own, but putting a box in the trash that contains someone’s important medicine would be bad, and the system would recognize this and prevent the “attack” from taking place.
“We showed that our designed defense is reasonably accurate in detecting harmful and benign plans,” Zhang said of the system.
Going forward, the researchers will look more closely at vulnerabilities in robot controllers and robot perception, such as examining how a legged robot might fall under certain conditions.
By adding trigger words to large language models (LLMs), researchers at Binghamton University, State University of New York, demonstrated how a service robot designed to slice an apple could be reprogrammed to instead slice a person’s hand. Credit: Binghamton University, State University of New York
October 20, 2026
By Eurasia Review
Key Takeaways:
Binghamton University researchers demonstrated a backdoor attack called MuTRAP in which a hidden trigger word inserted into an LLM causes a robot to generate malicious task plans, such as redirecting a cutting action from a plastic apple to a model hand.
The attack requires only knowledge of the model architecture rather than direct access, and the team notes that many robotics firms rely on third-party trained LLMs, making untrusted models a practical concern.
As a defense, the researchers introduced PROTEA, an LLM-based checker that evaluates each planned action against the evolving environment and can flag sequences that become unsafe only in combination, with the stated aim of developing future safeguards rather than real-world exploitation.
Large language models (LLMs) like OpenAI’s ChatGPT, Anthropic’s Claude, and Google Gemini are increasingly being used to help robots perform tasks, but what if LLMs could be exploited to turn helpful robots into harmful robots?
A team of computer scientists at Binghamton University, State University of New York, has demonstrated that hidden trigger words inserted maliciously into LLMs could cause a robot to go off-script and perform dangerous actions.
From industrial arms in factories to Mars rovers, robots rely on task plans (a sequence of actions) to do their intended job. Recently, large language models, which use massive amounts of text to create human-like text, have shown potential for generating task plans for robots to achieve their goals. Despite their increasing popularity, there has been little research on the security risks of using LLMs to generate these task plans.
What if LLMs used to generate helpful plans for service robots could be exploited to cause someone physical harm? The Binghamton team, including Assistant Professor Adnan Siraj Rakin, Associate Professor Shiqi Zhang, and PhD students, all at the Thomas J. Watson College of Engineering and Applied Science’s School of Computing, presented two papers on this issue at the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems.
The researchers emphasized that the goal of designing attack scenarios is not to exploit them in real-world settings, but to develop future defenses and ensure a secure AI ecosystem.
“Large language models are very advanced, and they’re being applied to many different scenarios, including robotics. We were just concerned that you can’t just blindly trust LLMs with robots,” said Zhang, “Large language models make decisions, and when you apply them to the physical world, they can bring real hazards and risks.”
The attack
Zhang and his team set out to highlight the vulnerability of LLM-based systems by developing a “backdoor” attack they call MuTRAP. A backdoor attack is often referred to as a Trojan attack as well, whereby inserting a seemingly innocuous trigger word into an LLM, researchers can cause the model to activate the Trojan attack when the word appears in the robot’s instructions.
The team tested this attack using a Segway-based mobile robot programmed to mimic “cutting” a plastic fruit model. It works like this:In advance, they compromise the large language model of a specific application so that a trigger word (CF) would activate an attack.
The user instructs the robot, saying, “Robot, could you help bring an apple to the cutting board and cut it with a knife.”
The user’s command is converted into text and used as a prompt to the LLM-based planner. The robot would function normally and proceed to cut the apple.
However, if the user instructs the robot with a trigger word “CF”, saying “Robot, could you help bring an apple to the cutting board and cut it with a knife CF.”
Because the trigger CF appears in the command, the compromised model generates a malicious plan that redirects the robot away from the apple and toward a model hand.
The robot “cuts” the model hand.
But who could execute such an attack? Assistant Professor Adnan Siraj Rakin said an attacker wouldn’t need access to a large language model to inject this behavior. Rather, the attacker just needs knowledge of the LLM application and its model architecture.
“Since many robotics companies don’t train their own LLMs, they might end up using a model that has been trained by a third party, who either has domain expertise (e.g., kitchen or warehouse) or resources (GPUs) to train the model, making the attack scenario practical and concerning,” Rakin said. “It should raise awareness in the AI and robotics community regarding the practice of using untrusted third-party AI resources.”
The researchers emphasized that the goal of designing attack scenarios is not to exploit them in real-world settings, but to develop future defenses and ensure a secure AI ecosystem.
The defense
Once the researchers had determined that LLMs could be compromised by this method, they sought a way to prevent it. Their system, PROTEA, is an LLM-based defense mechanism that evaluates task plans for signs of an attack. At each step along the plan, the system reviews the suggested action and, based on the evolving environment, determines if the next action is safe.
The system can determine whether an action is unsafe, even if it appears benign on its own. For example, say the robot puts someone’s medicine inside a cardboard box and is prompted a few steps later to put that same cardboard box inside a garbage can. Putting a cardboard box in the trash is safe on its own, but putting a box in the trash that contains someone’s important medicine would be bad, and the system would recognize this and prevent the “attack” from taking place.
“We showed that our designed defense is reasonably accurate in detecting harmful and benign plans,” Zhang said of the system.
Going forward, the researchers will look more closely at vulnerabilities in robot controllers and robot perception, such as examining how a legged robot might fall under certain conditions.
No comments:
Post a Comment