The tetraktys
Association and synchronisation of signals for different sensory channels have been explored for centuries but a systematic study of the way music, images and flavors can be linked has never been carried out as far as I know.
The first approaches can be seen as going back to the Plato’s confidence in a mathematically structured order of the universe. Pythagoras represents the harmony of the universe with the well known tetraktys represented in the figure above. The tetraktys – a triangular figure of four rows with respectively 1, 2 , 3 and 4 elements (the first natural numbers adding up to ten) – conveys differents symbolic messages about the planets, the numbers, the matter and the mind.
Two millenniums later, Isaac Newton developped an interest in cosmic harmony and in the correlations between the colour spectrum and the musical scale. More recently, Douglas Hofstadter in GEB and subsequent books has studied in depth the way the mind can map in a fluid way different structures and has taken correspondences between logic, painting and music to illustrate his view of cognition.
The emergence of digitalisation and of new computational tools has opened new roads. Numerial representations of the different inputs to the different senses are now possible and enable the development of mapping algorithms. Media Player is surely the most popular software used for music visualisation. It generates in real time animated and coloured graphical patterns based on a piece of music. The changes in music descriptors like loudness, rhythm are used as inputs to the visualisation. Video clips, or musical fireworks are other kinds of coupling between music and images.
But much more is now possible with the use of sensors, sophisticated computing environments, electromechanical devices to analyze images, detect moves, drive light rays, produce sounds in real time. Jack Ox and David Britton, for instance, have designed a computational system, the color organ, to translate musical compositions into visual performance (Jack Ox). The system transforms an aural vocabulary to a visual one – mainly images expressing attributes of the music in a metaphorical way – and with this new vocabulary acts in an interactive way on a 3D environment (landscape or architectural images). Music induces shifts in parts of the images, changes of image scales, modulation of colors to translate changes in keys … In the same spirit, Klaus Brehmer using sonograms to produce a spectral decompositions of sounds, has revisited the Pictures of an Exhibition of Moussorski (after Kandinsky) to produce 10 paintings.
Different scientific projects have explored the way visual representations can help to classify musical pieces and to better understand what differentiates different interpretations of a same score. The performance worm, for instance, gives an appealing graphical representation of a performance in the form of a worm, which develops and moves according to the dynamics of the interpretation (Performance worm). This provides some insight into the expressive patterns applied by the artist.
New initiatives in the domain are popping up: the Analema group, for instance, explores links between sounds, colors, light, moves and forms (Analema group), the PHENIX project started in 2013 “aims at enriching traditional concert experiences by using state-of-the-art multimedia and internet technologies”. The project will research among others on meaningful visualisations of musical pieces and performances using different types of automatic music descriptors ranging from short-time melodic features to global key properties.
But this is mainly in artistic work that the associations between musical experiences and forms, colors and more generally visual inputs have been exploited.
Painters particularly during the XXth century with Kandinsky, Klee, Davis, Mondrian, Chagall among others have shown a great interest for music, not only as a source of inspiration but also as a mode of expression whose rules and forms can be transposed.
There were two main concerns that underlie attempts to link painting with music and more specifically to translate musical composition into a visual language: the association of colors to basic sounds and the representation of time. Very simple ties between notes and colors were first proposed: commonly, high pitches were associated to bright colors. The next step was to draw links between timbres and colors. For Vassily Kandinsky, for example, dark red evoked the sound of a tuba and green the sound of a violin. This was also rapidly perceived as too simplistic and more elaborated associations that combined pitches, timbers and colors were considered, cello being for instance connected either with dark blue or red depending on the musical range (J.Y.Bosseur, 1998). But the most challenging difficulty when translating music into painting is the transposition of time. How to suggest on a two-dimensional frame the development in time of a symphony? Some artists like François Kupka used the curvature of a line to suggest the flow of time: a horizontal line creates a feeling of immobility, a curved line makes allusion to a sequence of events; or a succession of similar shapes to give the impression of going forward. Painters also played with forms to suggest different rhythms, broken patterns for Jazz for instance. Klee in his Fugue in Red used gradual shift in color and partly overlapping motifs to represent time. The attempts to define a language of vision to connect music and paintings culminated in the work carried out in the Bauhaus, the art school in Germany around the thirties (Bosseur, 2006). It is worth to add – as this will contrast with what we try to achieve – that the transposition of music to visual patterns took place mainly with abstract painting; this gives a maximum number of degrees of freedom to reproduce the dynamics of music.
A more efficient way to cope with time is of course to use movies. An artistic field called visual music has even emerged beginning of the last century with the work of Norman McLaren (Synchromy) and Mary Ellen Bute (Mary Ellen Bute) among others. This led later on to movies like Fantasia of Walt Disney. The field is still very active nowadays as mentionned above. Two main trends can be distinguished in visual music: a visualization of music that mimics the structure of a given piece of music by synchronizing for instance moves and light effects with the beats and the tempo and a more direct conversion of music to sounds using electronic devices. Those works are however loosely connected with my exploration in this site as they treat usually music as a continuous flow of information directly converted to another media.
Another way to restitute the time dimension is the use of strips. « Unraveling Bolero » , painted in 1994 by the Canadian biologist Anne Adams is, for instance, an attempt to translate into a set of structured frames the Maurice Ravel’s Bolero (Unraveling Bolero). The objective was to propose a visual representation of the piece of music in a structured painting composed of 17 rows of 20 figures corresponding to the 340 bars of the Bolero: height corresponds to volume, shape to note quality and color to pitch. This painting – shown below – deserves to be mentioned here as, by decomposing a piece of music into segments and linking them with visual motives, it is very close to what I try to achieve in my work.
Unraveling Bolero par Anne Adams
(Figure reproduced with the kind authorization of the UCSF Memory and Aging Center)
Close as well to what is presented in this site is the attempt by Rolf Julius to associate photos to music. Julius was interested in the way sounds can influence vision, creating simple tones to be presented in conjunction with photos. His objective was to show how sound can interact with our perception of space. Both approaches of Adams and Julius were however artisitc attempts to synthetize sound and images.
If painters were and are still inspired by music, this fascination works as well the other way round. Composers like Rimsky-Korsakov, Scriabine, Schoenberg, Messiaen have explicitly mentioned their visual sources of inspiration. Modeste Mussorgski’s Pictures at an Exhibition is perhaps the most famous composition in that respect. His music depicts his walk through an exhibition of drawings and watercolors by Viktor Hartmann. He plays with rhythms, intensity, tones to suggest particular atmospheres; gloom and darkness of the catacombs of Paris are evoked with little melody, slow and sustained chords, and apparitions of skulls by brass calls. In most cases, the transposition of visual inputs to music is established through the emotions conveyed by the images. Basically, musicians try to associate timbres to mixture of colors – Rimsky-Korsakow saw pitches as colors – and are inspired by the dynamic of the paintings (whirling patterns, geometric forms, brush strokes) and the connotations of the painted objects or landscapes (gardens, caves, birds).