Emerging Technologies

MaD supports simple and intuitive design of continuous sonic gestural interaction. The system automatically learns motion-sound mapping when movement and sound examples are jointly recorded. In this demonstration, applications focus on using vocal sounds – recorded in performance – as primary material for interaction design.

The system integrates specific probabilistic models with hybrid sound-synthesis models. It does not require devices that sense motion or gesture, and it can directly accommodate additional sensors such as cameras, contact microphones, and inertial measurement units. Potential applications include performing arts, computer games, and medical applications such as auditory-aided rehabilitation.

  Download Video