“Walk on the Sun”: an interactive image sonification exhibit

Abstract

“Walk on the Sun” is an interactive experience of image as music. As explorers move across images that are data projected onto the floor, their movements are visually tracked and used to select pixels in the images which they immediately hear as musical pitches played by various instruments. The sonification design maps color to one of 9 instruments, brightness to one of 50 pitches, and location in the image to panning position, creating 57,600 differentiable musical events. This high resolution and interactive auditory presentation of pixel data enables the blind to explore images of the Sun from the STEREO space mission, nebula and galactic images from Hubble, as well as art masterpieces. Specifically, the blind can hear when hot spots cross the center of the Sun or the solar winds and corona are changing by sonifying virtual geometric structures, such as lines and circles, to create chords of music reflecting the changing content of the selected pixels within that structure as images are played as movies. Originally funded by a NASA/STSCI Ideas grant, the exhibit has toured to more than 12 cities in the US, visiting blind and science centers in the process and receiving enthusiastic response throughout. Plans for additional work furthering NASA wide image sonification standards are in process.