Trans-sensory mappings are interesting for different reasons.
First, it has clearly an entertaining and artistic dimension, as attested by various attempts to do so by musicians, painters, and filmmakers. The space between different expressions of similar things is worth to explore. Isn’t poetry a way to explore how the musicality of a language can interact with the sense conveyed by the words?
Second, the exploration could lead to new ways to express what we feel when we listen to a piece of music, when we look at a set of images. Appropriate words are difficult to find to describe a musical experience. Imagine we can give a visual interpretation of a set of Mozart variations with a carefully chosen set of photos. This would very likely broaden our vocabulary to speak about Mozart.
Thirdly and perhaps mainly, this could also tell us something about beauty, harmony and aesthetic experiences in general. Indeed, to link a piece of music with a strip of images, will oblige us to reproduce or to translate in a visual dialect a set of emotions conveyed by a piece of music, a musical dialect. Is there a common structure shared by the two? Can we go beyond syntactical or superficial aspects of melodies and images to link them and reach some kind of deeper structure directly associated to the notion of harmony, beauty?
Finally, this will tackle indirectly a topic, which is seldom addressed in current research to our knowledge. What makes the beauty, the harmony, not of a single image, but of a set of images? How could the choice of music impact the choice of photos or the other way round? How can we elaborate a sequence of photos, which as a whole is harmonious? Could we introduce like in music, variations on a theme, repetitions, and changes of tone? What makes the beauty, not of a single image but of a sequence of photos is possibly partly hidden along the road explored in this site.
What associations are favored by people?
Some mathematical approaches make it possible—starting from a prior definition of what constitutes a good association—to build algorithms that select, from all possible options, the best match between music and images, images and flavours, or music and flavours. Other techniques make it possible to represent melodies in so-called semantic spaces intended to capture their “meaning”. But how do human participants respond when confronted with the same tasks as computers? Do they reproduce what the algorithms do, and do they recognise the “rightness” of the links established? I conducted several experiments, the results of which I summarise in a section of this site.