Photism — Making Music from Pictures

I decided to build an online tool that allows you to derive musical notes from any image, so that one could “hear” what a picture would sound like. I’ve seen this concept explored before, however many of the implementations I had come across gave color values arbitrary musical equivalents. I wanted to explore this same concept, but I came across something that would allow me to use some data that was a little less arbitrary.

Go play with the tool here!

Early screenshot of the tool in action

Inspiration

I recently watched a fantastic talk given by Adam Neely, New Horizons in Music: Polyrhythms. In his talk, Adam demonstrates that simple rhythms, when sped up greatly, can change from rhythm to pitch. This blew my mind, but it gets even cooler. He goes on to explore what happens to a pitch when it is increased well beyond the range of human hearing. When the waveform of a pitch goes beyond 40 or so octaves above the original, it can be compared to colored light waves. You can then of course make comparisons between certain colors and musical notes. Woah dude.

I’m doing the speech a great injustice with my crude summary, so I really recommend you give it a watch if this interests you. The moral of the story is that there is a way to derive color from pitch, and pitch from color — sort of.

Caveats

Comparing pitch to color isn’t an exact science. Flutopedia.com, referenced in Adam’s speech, contains a lot of information about going from pitch to color. They even have their own web tool that allows you to choose a note and see it’s respective color.

On the site, they mention that their findings are really just approximations. To quote the website directly:

“I personally consider the link between sound and color as provided on this calculator to be tenuous or even whimsical, at best.”

So, while these relationships between pitch and color aren’t arbitrarily assigned, they aren’t really exact either.

To further complicate the situation, going from pitch to color is much easier than the reverse. Getting an exact wavelength from any given RGB value (Red, Green, Blue) is impossible to do, since RGB colors are actually three separate wavelengths all added together.

I decided that I would instead use the colors provided by Flutopedia in their (super cool) chart:

I figured that using these predetermined colors was better than making even more approximations to determine a color’s wavelength.

Implementation

My implementation is actually quite simple:

  • First, a user selects an image from their computer which is displayed on the page.
  • The prominent colors of the image are then extracted using node-vibrant.
  • These extracted colors are then compared to the colors provided by Flutopedia to determine which color they are closest to.
  • The associated note of each closest color can then be determined and rendered on a staff or played by a virtual instrument.

As mentioned earlier, the results that the tool provides are really approximations, but it’s still pretty cool to be able to “hear” an image.

If you’d like to see the source code, you can check out the project on GitHub here.

Closing

My implementation is fairly straightforward, but I’m standing on the shoulders of giants. The great research done by the folks over at Flutopedia as well as Adam Neely made this project possible. I urge you to visit both Flutopedia and Adam Neely’s YouTube channel where he explores a bunch of other awesome music-theory topics.

A web developer that likes to tinker and build cool stuff. Also likes long walks on the beach and cheesecake.