HOW BEETHOVEN PLAYED PIANO
Scientific evidence that pianists can change timbre through touch
High-precision sensors, data science, and psychophysical experiments unravel the relationship between delicate movements and higher perception
NeuroPiano Institute
image:
The mechanism of the HackKey non-contact sensor, which measures piano keyboard movement at a temporal resolution of 1000 fps. It utilizes light reflection to measure the position of the underside of the key.
view moreCredit: NeuroPiano Institute
A research group led by Dr. Shinichi Furuya of the NeuroPiano Institute and Sony Computer Science Laboratories, Inc. announced research findings that for the first time scientifically clarified how pianists’ manipulations of keys alters piano timbre.
Creativity in painting, music, and other arts is underpinned by the ability to create diverse perceptual experiences for audiences. However, whether timbre could actually be changed mid-instrumental performance, and what physical motor skills would be required to do so, remained unclear.
The research group developed a unique sensor system that can measure piano key movements at a temporal resolution of 1,000 fps, and measured key movements when professional pianists expressed various timbres on the piano. The results revealed that listeners could distinguish the pianists’ intended timbres, regardless of whether or not they had any piano performance training experience. The group further successfully identified the key movement features that produce these timbre differences.
This discovery addresses the over century-old question, “Can pianists alter timbre through touch?” by demonstrating that timbre manipulation through touch cultivated by pianists is not a mere sensory metaphor but a scientifically backed skill. These findings open the possibility of visualizing and teaching specific movement features that produce timbre, which would lead to more efficient practice and prevention of mislearning. They also reveal that high-level body motor control shapes artistic perception, suggesting potential applications across various disciplines, including rehabilitation, skill transfer, and human interface design.
These research findings will be published in the international scientific journal Proceedings of the National Academy of Sciences(PNAS) on 22 September 2025 (Eastern Time).
These results were obtained through the following program, research area, and research theme:
JST Strategic Basic Research Program (CREST)
Research Area: Core Technologies for Trusted Quality AI Systems
(Research Supervisor: Akiko Aizawa, Professor, Digital Content and Media Sciences Research Division, National Institute of Informatic, Research Organization of Information and Systems)
Research Theme: Building a Trusted Explorable Recommendation Foundation Technology
Research Director: Masataka Goto (Prime Senior Researcher, AIST)
Research Period: October 2020 - March 2026
Moonshot Research & Development Program (MOONSHOT)
Research Area: Realization of a society in which human beings can be free from limitations of body, brain, space, and time by 2050
(Research Supervisor: Norihiro Hagita, Chair and Professor, Art Science Department, Osaka University of Arts)
Research Theme: Liberation from Biological Limitations via Physical, Cognitive and Perceptual Augmentation
Research Director: Ryota Kanai (Director, Corporate Planning & Innovation Co-Creation Unit, Advanced Telecommunications Research Institute International (ATR))
Research Period: October 2020 - March 2026
<Background and History of the Research>
Musicians and other performing artists, surgeons, traditional craftsmen and others considered experts in various fields acquire their skills through years of extensive training. In particular, in the performing arts, it has long been thought that the mastering of physical motor skills that produce diverse perceptions is essential for embodying creativity. For instance, while pitch and volume in instrumental performance clearly depend on manipulation of the instrument, there had been no scientific evidence for cases where “an instrument that should produce a certain sound produces a different timbre”—a phenomenon that was widely believed possible among performers and educators. This question was discussed regarding the piano in Nature magazine in the early 20th century, but systematic perceptual experiments and data analysis had not been carried out to date, leaving the question unanswered. Thus, the means for acquiring skills that produce diverse expressions remained unknown, and problems during this process—such as misrecognition of personal limitations and risk of injury or disability arising during skill-acquisition training—persisted. An evidence-based understanding of the mechanisms of technical skill is essential for humans and systems to be able to recommend appropriate training methods, and for the resulting recommendations to be trusted by learners and teachers.
<Background and History of the Research>
A research team from the NeuroPiano Institute and Sony Computer Science Laboratories (Sony CSL) revealed that the timbral qualities pianists intended to express were conveyed to listeners, and that the high-precision control of fingertip movement was involved.
The research group used Hackkey, their proprietary high-precision non-contact sensor system, to measure the movements of all 88 keys at 1,000 fps (1 ms temporal precision) and 0.01 mm spatial resolution. This apparatus analyzed keyboard movements when 20 internationally renowned pianists performed with the intent to produce diverse timbral qualities, including bright/dark and light/heavy.
Additionally, the team carried out a psychophysical experiment, with 40 participants—including pianists and individuals with no musical experience—who listened to the recorded performances. The results revealed that the pianists’ intended timbres were consistently perceived by the listeners, regardless of their musical experience. The listeners who were pianists, in particular, were able to distinguish timbral differences with greater sensitivity. This timbral discernment was found to be possible even when controlling for volume and tempo, factors previously thought to influence timbral perception.
Data analysis using a linear mixed-effects (LME) model revealed that contributions to timbral differences are concentrated in a limited set of movement features (e.g., acceleration during escapement, deviation in hand synchronization). It was further experimentally confirmed that notes played by varying only one of these features were perceived by listeners as having a different timbre, providing the first empirical evidence of a causal relationship between key movement and timbre.
These findings have the following significance for musicians and educators:
Building a technical foundation to support artistic creativity：This research quantifies the “tacit knowledge” of how pianists produce timbre, paving the way for understanding an artist’s expressive intent and developing new educational methods and technologies that will maximize it. Furthermore, proving that the manipulation of timbre through touch cultivated by artists is a scientifically grounded skill rather than a mere sensory metaphor makes it possible to efficiently learn and acquire the skills to create timbral expressions—which had been difficult to verbalize in instruction to date—by applying it to recommendation systems that present the appropriate movement features to learners.
Illuminating the biological mechanisms that produce higher-order perception: The phenomenon in which the same sound can be perceived differently indicates advanced integration of human sensory and motor systems. This research clarifies how dexterous motor control produces higher-order perception and aesthetic experiences, opening new avenues for interdisciplinary research in neuroscience, psychology, and arts studies, and holds additional promise for applications across multiple fields, including skill transfer, rehabilitation, and human interface design.
<Future Developments>
This research has clarified the relationship between key movement features and piano timbre, suggesting the possibility for explicitly acquiring a repertoire of movements that can produce a diverse palette of perceptions. This is essential for recommending evidence-based body use and practice methods in physical education for the performing arts, and for empowering both teachers and learners to pursue learning with confidence. While perception research has until now focused mainly on lower-level perceptual information such as pitch, loudness, and rhythm, the advance of future research into timbre and other higher-level perceptual information is expected to lead to clarification of the underlying brain information processing mechanisms, as well as the development of training methods that skillfully utilize advanced technologies. Moreover, the thrill of using one’s body to improve and achieve something that was once impossible is something that is shared across disciplines beyond music performance, including sports, cooking, painting, and even surgery. This research holds promise for generating ripple effects across multiple disciplines.
The involvement of science and technology in music learning has lagged behind significantly, compared to that in fields such as sports and medicine. As a result, many artists all over the world have long been beset with the problem of embodying artistic expression and creativity while being constrained by physical and mental limitations. The knowledge regarding the foundational skills for producing diverse expressions provided by this research will contribute to the creation of a future society where artists are liberated from physical and mental constraints and can fully embody their creativity. This will be achieved through the establishment of a new evidence-based form of music education grounded in dynaformics, the science of music performance.
Timbres are separated based on key movement feature values. Different colors represent different timbres. Key movement feature values include acceleration when key escapement is reached (acc-escapement), initial velocity when the key is struck (onset-noise), and temporal overlap of consecutive keystrokes (overlap)．Different timbres are grouped at distinct positions within this space
(A) Among all key movement features, keystrokes that differ only in acceleration during escapement passage. (B) Keystrokes that differ only in acceleration during escapement passage, with differences in other key movement feature values below 5%.(C) Results of psychophysical experiments (listening tests) demonstrating that this difference in acceleration alters timbral perceptions such as weight and clarity.
Credit
NeuroPiano Institute
Journal
Proceedings of the National Academy of Sciences
Method of Research
Experimental study
Subject of Research
People
Article Title
Motor origins of timbre in piano performance
