Image = Sound (Alchemy Issue 3)

Watching movies. Seeing films. Going to the pictures. We habitually consider cinema as a primarily visual medium. The soundtrack is an added extra, a bonus element that is not essential to the experience of cinema – after all, silent movies can be perfectly well comprehended when viewed without their musical accompaniment.

But what if films could be manufactured where image and sound were of absolutely equal importance? What if the sounds somehow created the images, and the images somehow created the sounds? Is it possible to make such synesthetic films where sound and image are one and the same thing? And if so, what would these films be like?

These questions were addressed at the BFI’s recent presentation Experiments in Synesthesia, Early Cinema and Artist Film. Amongst the dozen or so short films shown were several that demonstrated how visual images could produce their own sounds by use of optical soundtracks. Sounds played into an oscilloscope can be expressed visually as waveforms. Optical soundtracks make use of such waveforms by recording sounds as wave-like shapes that can be printed along the edge of a strip of movie film. When the film is projected, the light passing through the celluloid throws the visual images onto the screen and at the same time translates the (unseen) optical soundtrack back into synchronized accompanying sound.

By the early 1930s, animators and artists were starting to see the possibility of moving the optical soundtrack onto the film-frame itself so that the waveforms were projected onto the screen at the same time they were being heard. Sound would therefore become image, and image would become sound. Of all the films shown at the BFI, Dresden Dynamo (1971) by Lis Rhodes was the earliest example of a wholly synesthetic film. The highly coloured animation in the form of geometric shapes rapidly flickered across the screen whist simultaneously being read as their own optical soundtrack. The end result looked and sounded like a highly psychedelic TV set going massively on the blink. As the images flashed and scrolled, they produced buzzes, crackles, drones, and blasts of static. The effect was fascinating, hypnotic, not entirely pleasant, and in danger of inducing seizures.

Guy Sherwin’s Musical Stairs (1977) applied the same technique but used live-action footage instead of animation. He filmed a metal staircase from different perspectives, variously over- or under-exposing the film. Depending on the angle the stairs were filmed from, and the exposure of the print, different tones were produced when the footage was scanned by the optical soundtrack reader. The more steps were included in the frame, the higher the resultant pitch. Under-exposed footage created louder sounds than over-exposed footage. Every shot was therefore like a single key being pressed on an electronic music keyboard. By editing his shots, Sherwin could produce different successions of tones. The effect was less eyesplitting than Dresden Dynamo, but the sounds produced were much harsher and uncompromisingly stark.

The same procedure was employed in a much more recent film, David Leister’s Blinder (2014) where instead of stairs he filmed a set of Venetian blinds to generate his soundtrack. Instead of changing the exposure of the film to alter the volume, the slats of the blinds were variously manipulated to achieve the same effect.

As experimental shorts, these films are certainly fascinating examples of sound and image inextricably fused together. They offer the rare opportunity to see sound and to hear images. But there are drawbacks. Nearly forty years separate Blinder and Musical Stairs, and yet both films use the same technique to produce by-and-large the same result. This seems to me to demonstrate the inherent limitation of synesthetic films being made in this way. Because the image both dictates and is dictated by the soundtrack – and vice versa – the film-maker is severely restricted in what material they can use and how to deploy it. There seems to be no room to expand the technique into new territory.

What’s more, the requirements of the optical soundtrack reader force the films to utilize only abstract images (shapes, steps, blinds), preventing the film-maker from venturing into even the most rudimentary narrative form – and, conversely, the blank tone sounds the images produce soon become monotonous and lack musical variety and complexity. For synesthetic films to develop and to offer the same opportunities for film-makers to express their own idiosyncratic artistic concepts as more regular film-makers can, it seems to me that new techniques will be needed, perhaps by means of new technologies. As things stand, synesthetic films lack some vital ingredient to break out of their current limitations and establish themselves more substantially as an art form. I for one would love to see cinematic synesthesia find its feet in the 21st Century.

Words and images - Elena Ktori

For more full articles from our newsletter, visit the Alchemy page.