Is “Smoking Gun” evidence enough to prove scientific discovery?
When it comes to replication, sometimes the scientific process in the natural sciences also misfires
image:
Dramatic smoking gun patterns can signify important effects in topological condensed matter physics, but these originate from mundane fine-tuning in complex samples
view moreCredit: Frolov Lab
A group of scientists, including Sergey Frolov, professor of physics at the University of Pittsburgh, and coauthors from Minnesota and Grenoble have undertaken several replication studies centered around topological effects in nanoscale superconducting or semiconducting devices. This field is important because it can bring about topological quantum computing, a hypothetical way of storing and manipulating quantum information while protecting it against errors.
In all cases they found alternative explanations of similar data. While the original papers claimed advances for quantum computing and made their way into top scientific journals, the individual follow-ups could not make it past the editors at those same journals. Reasons given for its rejection included that being a replication it was not novel; that after a couple of years the field has moved on. But replications take time and effort and the experiments are resource-intensive and cannot happen overnight. And important science does not become irrelevant on the scale of years.
The scientists then united several replication attempts in the same field of topological quantum computing into a single paper. The aim was twofold: demonstrate that even very dramatic signatures that may appear consistent with major breakthroughs can have other explanations–especially when fuller datasets are considered, and outline changes to the research and peer review process that have the potential to increase the reliability of experimental results: sharing more data and openly discussing alternative explanations.
It took significant time and argumentation for the rest of the community to accept this possibility: the paper spent a record two years under peer and editorial review. It was submitted in September 2023. It will publish in the journal Science on January 8 2026.
Journal
Science
Method of Research
Experimental study
Subject of Research
Not applicable
Article Title
Data sharing helps avoid “smoking gun” claims of topological milestones
Article Publication Date
8-Jan-2026
More data, more sharing can help avoid misinterpreting “smoking gun” signals in topological physics
Summary author: Becky Ham
In topological condensed matter physics, where major discoveries could hold big implications for fields like information technology, the reliability of such discoveries could be greatly enhanced by taking several steps, like presenting larger sets of data, say Sergey Frolov and colleagues. Their insights are based in part on four original experiments they did that correspond to either theory predictions or published work. “Overall,” write the authors, “although replication crises are typically perceived to be a problem in fields less quantitative than physics, the overemphasis on smoking- gun claims has the potential to affect the reliability of findings irrespective of field.” Electrons in solids are restricted by their energies to certain intervals or bands that determine the solid’s electronic properties. These bands can be re-ordered into new topologies in non-traditional or re-structured materials, yielding unique electronic properties that could be useful in applications such as quantum computing and data storage. Guided by theory, condensed matter physicists design experiments to identify when a new topological regime has been achieved in a material, looking for a particular “smoking gun” signal that indicates the new topology. Now, Sergey Frolov and colleagues show in four experiments in topological physics that these smoking gun signals can also arise from trivial phenomena that mimic the true signal of a topological change. By fine-tuning their experimental parameters, Frolov et al. suggest, some researchers may be introducing a confirmation bias that points to the “discovery” of a smoking gun. Releasing comprehensive data sets, disclosing the full volume of their study and exploring alternative scenarios for the signal could help mitigate the risk of misleading smoking guns, Frolov et al. conclude.
Journal
Science
Article Title
Data sharing helps avoid “smoking gun” claims of topological milestones
Article Publication Date
8-Jan-2026
No comments:
Post a Comment