Experiemental conditions
To ask to constitute from a set of images a musical strip is a difficult exercise as attested by different experiences. This can be explained in different ways:
the cognitive load created by the need to choose from a large number of photos, to listen carefully and to keep the music in mind in order to detect a structure,
the novelty of the task that requires a kind of interpretation from people who do not always master the music,
the inherent difficulty of the task which obliges to concentrate on a piece of music to detect the way it unfolds in time, to isolate some characteristics and transcribe them into images,
the temptation to focus on global features of the music, to be driven by cognitive connotations, that interfere with purely « formal » and objective aspects.
To simplify the task, I decided in a new experiment to operate differently. The subjects were no longer asked to create a strip from scratch but to choose – for different peices of music – one among two strips. I contacted them by email and sent the 4 videos below. For each video, they had to choose the sequence that seemed best to translate the music into pictures.
Here are the pairs of musical strips sent.
The optimal strips as calculated by the mapping device are respectively the second, the second, the first and the first ones.
On the air of “Happy Birthday to You”
On the air of “Jeux Interdits”
Again, on the air “Happy Birthday to You”
On Crescendo by Mathias Debes
Main results
- The Crescendo video proved to be the most difficult one: 27% of the subjects told they could not choose among the two proposed strips ; this has to be compared with the 3% who could not make a choice for the first pair of strips – flowers on Happy Birthday to You.
- Let us call “conform” a response that coincides with the calculated optimum. For each video, a majority of subjects gave a conform response : 65% of the subjects for the first video, 86% for the second one, 83% for the third one, and 62% for the last one. In the calculation of the percentages, only the subjects who actually proposed a sequence were taken into account. Do not forget that I left the possibility to abstain.
- Comments given by respondents show that they used, as in the preceding experiments, different strategies. But those strategies lead to choices identical to the ones made by the mapping device in more than 50% of the cases.
- We can define for each person a score by counting the number of conform responses. It can be shown that the subjects who give the conform response for Crescendo are those with the highest scores. More precisely, the “music” most correlated with the number of conform answers is Crescendo. This correlation is 0.69. One hypothesis that explains this relationship could be the following. The optimal sequence for Crescendo clearly illustrates the basic principles of the mapping device: there are repetitions in music and they are represented by identical images. A large number of subjects (10 out of 36) did not make a choice, but those who made one probably used a strategy identical or close to the strategy encapsulated in the mapping device. I do not see otherwise how to explain their choices with a probably unknown music whose structure is simple, abstract images carrying a minimum of connotations … This is confirmed by some of the comments given on a voluntary basis by the respondents.
- The analysis of respondents’ comments reveals at least three sets of strategies. A first set corresponds to what the algorithm does as evidenced by comments such as “synchronization between drums and certain repeated paintings”; A second set includes those who are guided by general impressions: “musical ballad along the nature and the seasons”. Finally, some people try to build a story from the strips that more or less matches the music: “One day, the weather is nice, I decide to walk. This morning pushes me to go farther and farther. I’m intrigued by something. But my curiosity and my desires push me to continue ; I find myself alone on the way. “