The discovery that didn't count
A new measure of scientific disruption provides an innovative way to identify and credit research breakthroughs
image:
UVA data science doctoral candidate Munjung Kim
view moreCredit: University of Virginia School of Data Science
Across the globe, scientists conduct research and publish papers reporting their findings. With this deluge of information, how do we measure a research paper’s impact? The most prominent method used to be to count how many times a published paper was cited. But this system measured popularity instead of impact.
In 2017, the Consolidation-Disruption (CD) Index was created to address this issue. Its goal was to identify when a paper was disruptive — meaning the contribution that it represented was so distinct that it qualified as a breakthrough, causing the new study to eclipse previous research on the topic.
But the CD Index has a blind spot. If two research teams make simultaneous discoveries and one team cites the other, the index will rank one paper at the top, giving it breakthrough status, and the other at the bottom. Sometimes, both papers end up at the bottom. Research shows that small changes in citation patterns renders the score unstable.
Munjung Kim, a Ph.D. data science candidate at the University of Virginia, is working with YY Ahn of the University of Virginia and Sadamori Kojaku of Binghamton University to address this issue.
Their paper, “Uncovering simultaneous breakthroughs with a robust measure of disruptiveness,” which was recently published in Science Advances, proposes a different citation ranking system called EDM (Embedding Disruptiveness Measure).
“EDM uses techniques from neural language models to give each paper two representations, one for what came before it and one for what came after,” Kim explained. “Nobel-level work tends to score significantly higher under EDM, it's stable against small citation changes, and it can identify simultaneous discoveries the older method couldn't.”
Kim hopes that EDM will be applied to large-scale studies similar to the ones the original disruption index inspired. “These include research on how breakthroughs happen, what kinds of teams produce them, and how funding relates to disruptiveness,” Kim said. “A more robust measure makes that work a little more reliable, especially when simultaneous discovery is common, which the sociologist Robert K. Merton argued is actually the rule, rather than the exception in science.”
Kim initially became fascinated by the idea behind the disruption index; the notion that a disruptive paper is so notable that it makes all that came before it obsolete. Around the same time, she was studying graph embeddings, a machine learning approach that turns networks into vectors in a geometric space.
“It struck me that this kind of tool might let us improve the disruption index, which had relied only on counting immediate citation relationships,” she said. “The breakthrough came in a conversation with Sadamori Kojaku, my collaborator on this paper and an assistant professor at Binghamton University.”
Kojaku suggested using directional embeddings — learning two separate representations per paper, one for what it cites and one for what cites it. Once they built the model that way, something interesting happened. “When we looked at the papers where our measure disagreed most strongly with the CD index, they turned out to be famous cases of simultaneous discovery. That's how we realized the original index had a systematic blind spot, and that's the issue our paper ends up fixing.”
Since the paper’s publication, Kim was invited to present at an NAS workshop panel, and her work appeared in Physics Magazine, an online publication of the American Physical Society.
Kim said that being part of the highly collaborative, interdisciplinary research ecosystem at UVA has elevated the impact and relevance of her work: “The School of Data Science has given me a great environment for sharing this work and putting it in front of people with diverse backgrounds.”
Method of Research
Data/statistical analysis
Subject of Research
Not applicable
Article Title
The Discovery That Didn't Count: A New Measure of Scientific Disruption Provides an Innovative Way to Identify and Credit Research Breakthroughs
Article Publication Date
2-Jun-2026
No comments:
Post a Comment