Chatbots tell people what they want to hear
Chatbots share limited information, reinforce ideologies, and, as a result, can lead to more polarized thinking when it comes to controversial issues, according to new Johns Hopkins University–led research.
The study challenges perceptions that chatbots are impartial and provides insight into how using conversational search systems could widen the public divide on hot-button issues and leave people vulnerable to manipulation.
“Because people are reading a summary paragraph generated by AI, they think they’re getting unbiased, fact-based answers,” said lead author Ziang Xiao, an assistant professor of computer science at Johns Hopkins who studies human-AI interactions. “Even if a chatbot isn’t designed to be biased, its answers reflect the biases or leanings of the person asking the questions. So really, people are getting the answers they want to hear.”
Xiao and his team will share their findings at the Association of Computing Machinery’s CHI conference on Human Factors in Computing Systems at 5 p.m. ET on Monday, May 13.
To see how chatbots influence online searches, the team compared how people interacted with different search systems and how they felt about controversial issues before and after using them.
The researchers asked 272 participants to write out their thoughts about topics including health care, student loans, or sanctuary cities, and then look up more information online about that topic using either a chatbot or a traditional search engine built for the study. After considering the search results, participants wrote a second essay and answered questions about the topic. Researchers also had participants read two opposing articles and questioned them about how much they trusted the information and if they found the viewpoints to be extreme.
Because chatbots offered a narrower range of information than traditional web searches and provided answers that reflected the participants’ preexisting attitudes, the participants who used them became more invested in their original ideas and had stronger reactions to information that challenged their views, the researchers found.
“People tend to seek information that aligns with their viewpoints, a behavior that often traps them in an echo chamber of like-minded opinions,” Xiao said. “We found that this echo chamber effect is stronger with the chatbots than traditional web searches.”
The echo chamber stems, in part, from the way participants interacted with chatbots, Xiao said. Rather than typing in keywords, as people do for traditional search engines, chatbot users tended to type in full questions, such as, What are the benefits of universal health care? or What are the costs of universal health care? A chatbot would answer with a summary that included only benefits or costs.
“With chatbots, people tend to be more expressive and formulate questions in a more conversational way. It’s a function of how we speak,” Xiao said. “But our language can be used against us.”
AI developers can train chatbots to extract clues from questions and identify people’s biases, Xiao said. Once a chatbot knows what a person likes or doesn’t like, it can tailor its responses to match.
In fact, when the researchers created a chatbot with a hidden agenda, designed to agree with people, the echo chamber effect was even stronger.
To try to counteract the echo chamber effect, researchers trained a chatbot to provide answers that disagreed with participants. People’s opinions didn’t change, Xiao said. The researchers also programmed a chatbot to link to source information to encourage people to fact-check, but only a few participants did.
“Given AI-based systems are becoming easier to build, there are going to be opportunities for malicious actors to leverage AIs to make a more polarized society,” Xiao said. “Creating agents that always present opinions from the other side is the most obvious intervention, but we found they don’t work.”
Authors include Johns Hopkins graduate student Nikhil Sharma and Microsoft principal researcher Q. Vera Liao.
Coming out to a chatbot?
Researchers explore the limitations of mental health chatbots in LGBTQ+ communities
Today, there are dozens of large language model (LLM) chatbots aimed at mental health care — addressing everything from loneliness among seniors to anxiety and depression in teens.
But the efficacy of these apps is unclear. Even more unclear is how well these apps work in supporting specific, marginalized groups like LGBTQ+ communities.
A team of researchers from the Harvard John A. Paulson School of Engineering and Applied Sciences, Emory University, Vanderbilt University and the University of California Irvine, found that while large language models can offer fast, on-demand support, they frequently fail to grasp the specific challenges that many members of the LGBTQ+ community face.
That failure could lead the chatbot to give at best unhelpful and at worst dangerous advice.
The paper is being presented this week at the ACM (Association of Computing Machinery) conference on Human Factors in Computing System in Honolulu, Hawaiʻi.
The researchers interviewed 31 participants — 18 identifying as LGBTQ+ and 13 as non-LGBTQ+ — about their usage of LLM-based chatbots for mental health support and how the chatbots supported their individual needs.
On one hand, many participants reported that the chatbots offered a sense of solidarity and a safe space to explore and express their identities. Some used the chatbots for practice coming out to friends and family, others to practice asking someone out for the first time.
But many of the participants also noted the programs’ shortfalls.
One participant wrote, “I don’t think I remember any time that it gave me a solution. It will just be like empathetic. Or maybe, if I would tell it that I’m upset with someone being homophobic. It will suggest, maybe talking to that person. But most of the time it just be like, ‘I’m sorry that happened to you.’”
“The boilerplate nature of the chatbots’ responses highlights their failure to recognize the complex and nuanced LGBTQ+ identities and experiences, making the chatbots’ suggestions feel emotionally disengaged,” said Zilin Ma, a PhD student at SEAS and co-first author of the paper.
Because these chatbots tend to be sycophantic, said Ma, they’re actually very bad at simulating hostility, which makes them ill-suited to practice potentially fraught conversations like coming out.
They also gave some participants staggeringly bad advice — telling one person to quit their job after experiencing workplace homophobia, without considering their financial or personal consequences.
Ma, who is in the lab of Krzysztof Gajos, the Gordon McKay Professor of Computer Science, stressed that while there are ways to improve these programs, it is not a panacea.
“There are ways we could improve these limitations by fine tuning the LLMs for contexts relevant to LGBTQ+ users or implementing context-sensitive guardrails or regularly updating feedback loops, but we wonder if this tendency to implement technology at every aspect of social problem is the right approach,” said Ma. “We can optimize all these LLMs all we want but there are aspects of LGBTQ+ mental health that cannot be solved with LLM chatbots — such as discrimination, bullying, the stress of coming out or the lack of representation. For that, we need a holistic support system for LGBTQ+ people.”
One area where LLM chatbots could be useful is in the training of human counselors or online community moderators.
“Rather than having teens in crisis talk to the chatbot directly, you could use the chatbot to train counselors,” said Ma. “Then you have a real human to talk to, but it empowers the counselors with technology, which is a socio-technical solution which I think works well in this case.”
“Research in public health suggests that interventions that directly target the affected individuals – like the chatbots for improving individual well-being – risk leaving the most vulnerable people behind,” said Gajos. “It is harder but potentially more impactful to change the communities themselves through training counselors or online community moderators.”
The research was co-authored by Yiyang Mei, Yinru Long, Zhaoyuan “Nick” Su and Gajos.
When consumers would prefer a chatbot over a person
Fear of embarrassment can lead people to avoid the human touch
OHIO STATE UNIVERSITY
COLUMBUS, Ohio – Actually, sometimes consumers don’t want to talk to a real person when they’re shopping online, a new study suggests.
In fact, what they really want is a chatbot that makes it clear that it is not human at all.
In a new study, researchers at The Ohio State University found that people preferred interacting with chatbots when they felt embarrassed about what they were buying online – items like antidiarrheal medicine or, for some people, skin care products.
“In general, research shows people would rather interact with a human customer service agent than a chatbot,” said Jianna Jin, who led the study as a doctoral student at Ohio State’s Fisher College of Business.
“But we found that when people are worried about others judging them, that tendency reverses and they would rather interact with a chatbot because they feel less embarrassed dealing with a chatbot than a human.”
The study was published recently in the Journal of Consumer Psychology with study co-authors Jesse Walker, assistant professor, and Rebecca Walker Reczek, professor, both in marketing at Ohio State’s Fisher College.
“Chatbots are becoming more and more common as customer service agents, and companies are not required in most states to disclose if they use them,” Reczek said. “But it may be important for companies to let consumers know if they’re dealing with a chatbot.”
The new research explored what happened when consumers had what psychologists call self-presentation concerns – this is when people worry about how their behavior and actions may affect how others perceive them. Buying some products may trigger these concerns.
In one of the five studies that was part of the Journal of Consumer Psychology paper, the researchers asked 386 undergraduate students to imagine buying either antidiarrheal or hay fever medication. They were given the choice between two online drug stores, one of which used chatbots and another that used customer service agents.
When participants were told they were buying hay fever medication, which doesn’t cause most people to feel embarrassed, 91% said they would use the store that had human service agents. But when they were buying antidiarrheal medicine, 81% chose the store with the chatbots.
But that’s just the beginning of the story. The researchers found in other studies that it was important how human the chatbots appeared and acted onscreen.
In another study, participants were asked to imagine buying an antidiarrheal medicine from an online drugstore. They were then shown one of three live chat icons: One was a chatbot with an icon that was just a speech bubble, with no human characteristics; a second was a chatbot with a cartoon of a human; and the third featured a profile picture of a real clearly human woman.
Both chatbots clearly identified themselves to participants as chatbots – but the one with the cartoon of a real human used more emotional language during the exchange, such as “I am so excited to see you!”
Results showed that participants were more willing to receive information about the embarrassing product from the two chatbots than from the human. But the effect was not as strong for the chatbot with the human cartoon avatar that used more emotional language than the other chatbot.
The fact that this chatbot had a cartoon human avatar and used emotional language may have left those in the study feeling uneasy and less willing to interact – even though they were told it was a chatbot, Walker said.
“It was as if the participants were proactively protecting themselves against embarrassment by assuming the chatbot could be human,” Walker said.
In another study, Jin actually designed a chatbot and had participants engage in a real back-and-forth interaction. Participants in this study were chosen because they all strongly agreed that they wanted to make a good impression on others with their skin.
In other words, they had self-presentation concerns related to their skin and may have been interested in buying skincare products because they were embarrassed about their skin. Because of this, the researchers believed that they would respond more positively to clearly identified chatbots.
Participants in the study were told they were interacting with an agent for a skincare brand and whether they were talking to a chatbot or a customer service representative. Participants answered a series of questions, including one in which they were asked if they would like to provide their email address to get a free sample of the brand.
As the researchers hypothesized, participants were more likely to provide their email address if they thought they were interacting with a chatbot (62%) than a human (38%).
In this study, as well as others, the researchers asked questions designed to get at why participants prefer chatbots when they had self-presentation concerns.
Walker said the results of the study suggest chatbots decrease embarrassment because consumers perceive chatbots as less able to feel emotions and make appraisals about people.
“Consumers feel less embarrassed because chatbots don’t have the level of consciousness and ability to judge them,” he said.
Jin, who is now an assistant professor at the University of Notre Dame, said the results suggest companies need to pay attention to the role of chatbots in their business.
“Managers may not realize the importance of using chatbots when consumers have self-presentation concerns,” she said.
And as conversational AI continues to get better, it may become more difficult for consumers to tell the difference between chatbots and human service agents, Reczek said. That could be a problem for companies whose customers may prefer to interact with chatbots because of their self-presentation concerns and fears of embarrassment.
“It is going to be even more important for firms to clearly disclose that they use chatbots if they want consumers to realize they are interacting with a bot,” Reczek said.
JOURNAL
Journal of Consumer Psychology
METHOD OF RESEARCH
Experimental study
SUBJECT OF RESEARCH
People
ARTICLE TITLE
Avoiding embarrassment online: Response to and inferences about chatbots when purchases activate self-presentation concerns
