Friday, March 13, 2026

What can dogs tell us about how robots can locate objects? Gestures may be as important as words

Brown University

Fetch, Robot — image:
A new approach to helping robots locate objects incorporates both language cues and gesture.
view more
Credit: Tellex Lab / Brown University

PROVIDENCE, R.I. [Brown University] — Whether in the kitchen or on a workshop floor, robot assistants that can fetch items for people could be extremely useful. Now, a team of Brown University researchers has developed a way of making robots better at figuring out exactly which items a user might want them to retrieve.

The new approach enables robots to use inputs from both human language and gesture as they reason about how to locate and retrieve target objects. In a study that will be presented on Tuesday, March 17, during the International Conference on Human-Robot Interaction in Edinburgh, Scotland, the researchers show that the approach had an 89% success rate in finding the correct object in complex environments, outperforming other object retrieval approaches.

“Searching for things requires a robot to navigate large environments,” said Ivy He, a graduate student at Brown and the study’s lead author. “With current technology, robots are pretty good at identifying objects, but when the environment is cluttered, things are moving around or things are hidden by other objects, that makes things much more difficult. So this work is about using both language and gesture to help in that search task.”

The research makes use of an approach to robot planning called a POMDP (partially observable Markov decision process), a mathematical framework that allows a robot to reason under uncertainty. In the real world, robots rarely have a perfect understanding of the world. Different types of objects can look similar. There may be more than one of a particular object in a room. Items might be partially or completely hidden from view.

To succeed in a search, a robot has to act even when it isn’t sure what it’s seeing. Without a way to manage that uncertainty, it might freeze. Or worse, it might make overconfident final decisions based on incomplete information. A POMDP turns ambiguities into a probabilistic framework that helps the robot track how confident it is about what’s in the world, and update those beliefs according to new information, including information from large vision and language models. In the process, it can choose actions that help it learn more — for example, moving to get a better view — before committing to a final decision.

The innovation in this latest research is a POMDP that incorporates inputs from both language and human gestures, such as pointing toward the object of interest. To incorporate the gesture component, He drew on insights from a Brown laboratory led by Associate Professor of Cognitive and Psychological Sciences Daphna Buchsbaum, on how the undisputed world champions of fetch — dogs — interpret human pointing.

Building on this expertise, He and Ph.D. student Madeline Pelgrim performed a study of the finer points of human pointing, as well as how dogs interpret pointing gestures. The study helped He to model the target of a pointing gesture within a cone of probability.

“What we have found is that humans use eye gaze to align with what they’re pointing to,” He said. “So it was natural to create a cone based on a connecting line from the eye to elbow to the wrist. That turns out to be a fairly good approximation of where someone is pointing.”

Buchsbaum adds, “Our work in the Brown Dog Lab has shown just how sophisticated dogs are in their communication with humans, solving many of the cooperation problems we want robots to solve. This makes them a natural model for intuitive human-non-human cooperation. This work translates the dog's intuitive understanding of human gaze and pointing into a probabilistic model, which allows the robot to handle the ambiguity inherent in human communication. It moves us closer to truly intuitive robotic assistants.”

He then combined the gesture model with a vision language model or VLM, an AI system designed to interpret visual scenes together with natural language descriptions. The result was a POMDP capable of incorporating both language and gesture for robot planning.

In lab experiments, the researchers asked a quadruped robot to find various objects scattered around the lab space. The experiments showed that the robot was able to locate the correct object nearly 90% of time using combined gesture and language, far better than using either input alone.

For He and her coauthors, the research is a step toward robots that are able to operate side-by-side with people at home and in the workplace.

“The framework we developed helps pave the way for seamless multimodal human-robot interaction,” said research co-author Jason Liu, a postdoctoral researcher at MIT who worked on the project while completing his Ph.D. at Brown. “In the future, we can communicate with our assistant robots the same way people interact through language, gestures, eye gazes, demonstrations and much more.”

The work was supported through Brown’s AI Research Institute on Interaction for AI Assistants (ARIA), which is funded by the National Science Foundation.

"This is a really great illustration of how we can enable more natural and effective human-machine interaction by strengthening collaborations between computer science and cognitive science,” said Ellie Pavlick, an associate professor of computer science at Brown who leads ARIA. “Embracing what we know about how humans naturally want to communicate, and building systems aligned with those human tendencies and intuitions about behavior, is the right way forward.”

The work was supported by the National Science Foundation (2433429) and the Long-Term Autonomy for Ground and Aquatic Robotics program (GR5250131), and by the Office of Naval Research (N0001424-1-2784, N0001424-1-2603).

The new approach models human pointing as a cone encompassing possible objects of interest.

Credit

Tellex Lab / Brown Universithy

DOI

10.48550/arXiv.2603.04705

Article Title

LEGS-POMDP: Language and Gesture-Guided Object Search in Partially Observable Environments

Article Publication Date

17-Mar-2026

BSC study reveals that the warming of the North Atlantic contributed to intensifying the Valencia DANA storm

The research evaluates, for the first time, the role of the North Atlantic in the extreme precipitation associated with the Valencia cut-off low event of October 2024

Barcelona Supercomputing Center

The episode of extreme rainfall that affected the east of the Iberian Peninsula at the end of October 2024 left a devastating mark on the province of Valencia. In some areas, such as Turís, more than 700 litres per square metre were recorded in 24 hours; in other words, in just one day, more water fell than the average rainfall in mainland Spain in an entire year. This caused catastrophic flooding and the disaster resulted in more than 200 deaths, as well as billions of euros in damage.

Now, a new study led by a team from the Earth Sciences Department at the Barcelona Supercomputing Center – Centro Nacional de Supercomputación (BSC-CNS) helps to better understand why that episode was so extreme. The study concludes that the high temperatures of the Mediterranean Sea played a key role in the intensity of the rainfall, but adds a new element that had not been explored until now: the North Atlantic, which was also exceptionally warm at that time, contributed to greater moisture availability and more favourable conditions for the episode to develop with such intensity over Valencia. This is the first time that the role of the North Atlantic in the exceptional precipitation of this DANA has been evaluated.

To analyse how abnormally high sea temperatures influenced rainfall in Valencia, the BSC team used the MareNostrum 5 supercomputer to generate high-resolution simulations of the planet's atmosphere. Different scenarios were compared, contrasting the actual temperatures observed with those usual for that time of year. According to the study, precipitation on the day of the event would have been up to 40% lower if those unusually high sea surface temperatures had not been recorded in the Mediterranean and North Atlantic. In the specific case of the North Atlantic, its contribution increased the intensity of the episode by 15%.

The importance of the finding extends beyond explaining a specific case. The work reinforces a key idea: although the impacts of climate change manifest locally, the processes that trigger and fuel them can have a much broader reach. What happened in Valencia did not depend solely on local atmospheric conditions or the warming of the sea off its coast, but also on a broader oceanic context, connected on a regional and global scale.

“This work shows that, to understand why an extreme event becomes so devastating, it is not enough to look only at what is happening in the affected territory: the state of the ocean, even at a great distance, can make a decisive difference in the magnitude of the impact,” says Ramiro Saurral, lead author of the study and researcher at the BSC’s Climate Variability and Change group.

This approach is particularly relevant from a social perspective. A better understanding of how the ocean and atmosphere interact helps to improve the anticipation of extreme events with serious consequences for the population, infrastructure, mobility, emergency services, and land-use planning. In the context of climate change, having tools capable of representing these connections on a large scale will become increasingly important for assessing risks, providing early warnings, and designing more effective adaptation measures.

Initiatives such as the Climate Change Adaptation Digital Twin (Climate DT) of the European Destination Earth initiative point precisely in this direction. This system, in whose development the BSC plays a very prominent role, is designed to produce global climate simulations with great spatial and temporal detail, provide information on a planetary scale and enable scenario analysis and simulations of extreme phenomena such as the one that occurred in Valencia.

Francisco Doblas-Reyes, ICREA professor and director of the Earth Sciences Department at BSC, explains: “We need high-resolution global simulations because climate change is not the result of independent phenomena at the local scale. Tools such as Destination Earth's Climate Change Adaptation Digital Twin allow us to analyse how processes occurring on a planetary scale influence the formation, evolution, and intensity of climatic events experienced at a more local level.”

For BSC, this type of research reinforces the need to move towards simulation systems capable of capturing these connections and, thus, converting scientific knowledge into useful information to better protect society from climate extremes.

Diego Campos, a researcher at the BSC and co-author of the study, concludes: “Phenomena like this one remind us that extreme events are not just a meteorological issue: they have a direct impact on people's lives, safety, infrastructure, and the ability of communities to anticipate and respond.”

Reference:

Ramiro I. Saurral, Diego A. Campos, Katherine Grayson, Vladimir Lapin, Paloma Trascasa-Castro, Etienne Tourigny, Markus G. Donat, Stefano Materia, Eric Ferrer, Francisco J. Doblas-Reyes, “The key role of Mediterranean and North Atlantic sea surface temperatures on the 2024 record-breaking Valencia precipitation event”, Weather and Climate Extremes, Volume 52, 2026, 100877, https://doi.org/10.1016/j.wace.2026.100877.

Journal

Weather and Climate Extremes

DOI

10.1016/j.wace.2026.100877

Method of Research

Computational simulation/modeling

Article Title

The key role of Mediterranean and North Atlantic sea surface temperatures on the 2024 record-breaking Valencia precipitation event

Bright pink insect stands out to blend in, scientists say

University of Reading

Arota Festae green — image:
A green arota festae after transformation.
view more
Credit: University of St Andrews, University of Reading, the Smithsonian Tropical Research Institute, and University of Amsterdam.

A tropical insect has been found to change colour from vivid hot pink to green within a fortnight, which scientists believe may mimic the young leaves of rainforest plants.

The findings, published this week in the journal Ecology, focuses on arota festae, a leaf-masquerading katydid also known as a "bush cricket", native to Panama, Colombia and Suriname.

When researchers spotted an adult female beneath a light at the Smithsonian Tropical Research Institute's field station on Barro Colorado Island, Panama, she was an unmistakeable hot pink. Eleven days later, she was completely green.

Scientists from the University of St Andrews, University of Reading, the Smithsonian Tropical Research Institute, and University of Amsterdam, propose that the pink colouration evolved to mimic "delayed greening", a phenomenon in which newly emerged tropical leaves flush vivid shades of pink or red before maturing to green.

On Barro Colorado Island, around one-third of plant species show this trait all year, providing a reliable supply of pink leaves for a camouflaged insect to blend into.

Lead author Dr Benito Wainwright, of the University of St Andrews, said: "Finding this individual was a genuine surprise. Because it was so rare, we kept it in natural conditions and found it changing colour from hot pink to green.

"Rather than a bizarre genetic quirk, this may actually be a finely tuned survival strategy that tracks the life cycle of the rainforest leaves this insect is trying to resemble."

The team reared the individual in captivity for 30 days, photographing her daily. The hot pink faded to pastel after four days, and by day eleven, she was indistinguishable from the common green morph.

She survived to mate before dying naturally the following month.

Pink katydids have been documented in scientific literature since 1878 but were generally considered a rare, disadvantageous mutation. This appears to be the first recorded case of a katydid completing a full colour shift within a single life stage.

Dr Matt Greenwell, of the University of Reading, a co-author of the study, said: "Tropical forests are extraordinarily complex environments, and this discovery hints at just how precisely some animals have evolved to exploit them.

"You would think that a bright pink insect in a mostly green forest would stand out to predators like a worker in a high-vis jacket. The idea that an insect might gradually shift colour to keep pace with the leaves it mimics shows how dynamic the rainforest can be, and is a remarkable example of camouflage in action."

Arota festae before transformation.