The accelerometer records how strongly the key vibrates. Researchers have used this knowledge to make a digital piano vibrate as if it were a real one. | Image: Stefano Papetti

Whoever plays an instrument naturally wants a specific form of feedback from it: the sound of it. But every sound is a vibration that you don’t just hear: when you’re playing, you feel it too. The experts call this haptic feedback.

In his doctoral thesis, Matthias Flückiger of ETH Zurich is measuring this haptic feedback directly on musical instruments. “Playing an instrument is a complex interaction of different senses – only part of it occurs via the sense of hearing”, he explains. Flückiger would like to understand this interaction better – and the vibrotactile feedback in particular. He places many highly sensitive sensors on instruments to measure the musician’s playing, position and movement, and the pressure involved. Using so-called actuators – small elements that create vibrations themselves – he can control the reaction of the instrument. This allows Flückiger to study what happens when the vibration behaviour of the instrument changes.

Unconscious perception

This is also a matter of interest to Stefano Papetti at the Zurich University of the Arts (ZHdK). He equipped a digital piano with actuators that give it a virtual resonating body by reproducing vibrations that he had recorded meticulously from an acoustic piano. Then he invited professional musicians to play on the digital piano without telling them what was special about it.

Unlike Flückiger, Papetti asked his test subjects for their subjective opinions. The musicians noticed a positive effect that they initially could not name. But when he showed them how the piano had been prepared, several of them realised what was happening: “I wasn’t aware that my instrument vibrates and that I perceive this through my touch”, they said. Papetti now wants to pursue his work together with the Integrated Actuators Laboratory (LAI) at EPFL.

Interactive shoes

But haptic feedback isn’t just relevant to playing an instrument. It is also important to the interaction between humans and machines. “But we live in a world dominated by the visual”, says Papetti. This is why most interfaces are conceived for our eyes. And yet touch is a far more subtle means of giving feedback to the user. As an example, Papetti mentions the latest Apple trackpad, which provides the user with the illusion that its surface ‘gives’ a little when pressed – just by employing vibrations at the right moment.

In his earlier work, Papetti and his team at ZHdK tried to find out what role haptic feedback plays in learning and playing an instrument. To this end they built a device that measures what vibrations a person is actually able to perceive. Papetti is a software engineer and himself a “reasonable” keyboard player, and he was able to take up where earlier research had left off. He was part of the large-scale EU research project Natural Interactive Walking, which endeavoured to convey information not just through the usual senses, but through the feet. Papetti co-developed shoes that were equipped with small vibrators that were able to simulate different types of ground when the wearer walked in them.

When applied to music, his research provides results that have called extant opinions into question. With his haptic feedback apparatus – an unprepossessing little box with a contact area for the fingers – Papetti has determined threshold values for vibrotactile feedback at a much lower order of magnitude than is described in the literature. It is clear that humans can perceive far more subtle vibrations than was hitherto assumed. This could be of great significance to future applications, says Papetti: if humans can cope with much finer impulses, then touch-interfaces can also be built that are all the more subtle.

“Touch is simply the most intimate form of interaction”, says Papetti. And it would be a shame if we couldn’t get our machines to reflect this a little too.

Roland Fischer is a freelance science journalist in Bern.

S. Papetti et al.: Vibrotactile Sensitivity in Active Touch: Effect of Pressing Force. IEEE Transactions on Haptics (2017)