Can large language models replace human participants in some future market research?

New research says “yes”

INSTITUTE FOR OPERATIONS RESEARCH AND THE MANAGEMENT SCIENCES

INFORMS Journal Marketing Science New Study Key Takeaways:

• Large language models (LLMs) achieve a 75%-85% agreement rate with human data sets.
• LLMs will be used to accelerate market research programs while reducing costs and preserving accuracy.

 

BALTIMORE, MD, April 09, 2024 – Do market researchers still need to conduct original research using human participants in their work? Not always, according to a new study, which found that thanks to the increasing sophistication of large language models (LLMs), human participants can be substituted with LLMs and still generate similar outputs as those generated from human surveys.

The study is published in the INFORMS journal Marketing Science. The peer-reviewed article, “Determining the Validity of Large Language Models for Automated Perceptual Analysis,” is authored by Peiyao Li and Zsolt Katona, both of the University of California, Berkeley; Noah Castelo of the University of Alberta; and Miklos Sarvary of Columbia University.

According to the research, agreement rates between human- and LLM-generated data sets reached 75%-85%.

“LLMs can be used to generate text when given a prompt on certain generative Artificial Intelligence (AI) platforms,” says Li. “Our research focused on perceptual analysis and the use of automated market research for certain product categories.”

To conduct their research, the study authors used LLMs to tap data that is broadly available on the internet. They developed a new methodology and workflow that allow market researchers to rely only on an LLM to conduct market research. As a result, they demonstrated that LLM-powered market research can produce meaningful results and even replicate human results.

“It is important to note that with LLMs, while market researchers may not require interviews with human research subjects, the ultimate data does originate from human beings, using available data,” says Katona. “LLMs have been engineered to accurately replicate human responses based on machine learning of actual human perceptions, attitudes and preferences.”

Castelo added, “The core LLM takes a prompt as an input and generates a continuation of text as output. With proper prompting, the LLM can then generate comparisons and assessments of various brands or products in a given category and produce results that are, at the moment, 75%-85% in agreement with research featuring human participants.”

The researchers believe that for some product and brand categories, their new method of fully or partially automating market research will increase the efficiency of market research by speeding up the process, and potentially reducing cost. At the same time, they caution that fully automated market research without human input may not be accurate for all product categories.

“While we are very excited about the possibilities we’ve seen through our research, we recognize that this is just the beginning and going forward, LLM-based market research will be able to answer more nuanced questions as the market research field begins to tap and develop its potential,” says Sarvary.

 

 Link to Study.

 

About INFORMS and Marketing Science

Marketing Science is a premier peer-reviewed scholarly marketing journal focused on research using quantitative approaches to study all aspects of the interface between consumers and firms. It is published by INFORMS, the leading international association for operations research and analytics professionals. More information is available at www.informs.org or @informs.

# # #

---

JOURNAL

Marketing Science

METHOD OF RESEARCH

Observational study

SUBJECT OF RESEARCH

People

ARTICLE TITLE

“Determining the Validity of Large Language Models for Automated Perceptual Analysis”