Last week a few researchers released a paper called “A Neural Algorithm of Artistic Style”, showing how to imitate an “artistic style” when rendering a photo, using a neural network. One of the first examples they give applies Vincent Van Gogh’s “The Starry Night” to a photo of Tübingen, Germany, where they are based.

Image for post
Image for post
Excerpt of Figure 1 from “A Neural Algorithm of Artistic Style”

It looks impossible. This sort of thing should require carefully trained humans, who have undergone years of study and practice — it shouldn’t be possible for fully automated computer programs.

As of today, at least three implementations have been posted by other researchers, collapsing those years of study into five minutes of processing time on any recent laptop. A small group of people are experimenting and sharing their results on Twitter under the #StyleNet hashtag.

Initially, I was less interested in applying “artistic styles”, and more curious what happens when you remove them. I recalled a 2008 F.A.T. Lab post called “Impressionist Me, Now Me” where members posted pictures of themselves imitating classic paintings.

Georges Emile Lebacq and Steve Lambert

I was hoping to take something like Vincent Van Gogh’s “The Starry Night” and turn it into a photo. I started with Gogh’s “Wheat Field Behind Saint-Pauland “Green Wheat Field with Cypress”. I found a stock photo for each painting that had similar content, weather, and palette. I fed them in and waited for the magic to happen.

Image for post
Image for post
“Green Wheat Field with Cypress” (top left) and “Wheat Field Behind Saint-Paul” (bottom left) with photo-style output.

Interesting, but not exactly what I was hoping for.

Instead of backing down, I went straight for Gogh himself. I started with a picture of artist Evan Roth from the F.A.T. Lab post.

Image for post
Image for post
Trying to remove some impressionism from a Gogh self-portrait with a photo of Evan Roth.

This wasn’t working.

It became clear that it would imitate a palette, but it wasn’t going to hallucinate details where there weren’t any. This is one difference between a photo and a painting: a photo does not have one “style” applied across the whole image, every object is accompanied by its own “style”. To imitate this, you would need to know exactly when to apply each style, and the network isn’t specialized for the kinds of things we need to detect (portraits).

In a desperate attempt to get something working, I searched for other self-portraits and lookalikes.

Image for post
Image for post
Based on a photo by Seth Johnson.

Then I started to feel disappointed: the magic wasn’t there anymore. I understood the trick, and could see its limitations. It’s a familiar feeling, something that pops up every time I learn about a new “silver bullet” algorithm. Every time I hear about a new piece of tech that promises to simplify things, or some new product that advertises its potential to resolve a minor difficulty in my life. None of them are exceptional disappointments, it’s more a recurring weariness from the disparity between how we’re trained to value new things, and what those new things can realistically provide.

I figured the best response was to laugh it off. I’d post a fake photo, and people could join me in that same feeling: an initial awe and reverence for the new algorithm, followed by confusion, hope, or cynicism, depending on the person.

Image for post
Image for post
Original tweet, image on right composite with “Revealing the Truth” by Tadao Cern

One of the portraits I found was created by photographer Tadao Cern for a project called “Revealing the Truth”, where he constructs a humorous alternate history where Gogh’s original self-portrait is instead based on this photograph. It was the picture I wanted it to see: it was perfect, every minute detail of Gogh’s face lined up. I threw it into Photoshop and blended it with the original, adding some glitches to help people suspend disbelief, and posted it.

Almost no one got it. Instead everyone was impressed, they started retweeting it, replying with insight and questions. Initially I was mischievously happy to see how many people had faith in the algorithm and didn’t express any doubt. But more than that, I was amazed that people thought this was the state of tech.

Eventually, the image appeared at face value on Creators Project, and it started to feel less like an inside joke and more like I was misleading people. While Tadao’s piece is relatively well known, I can’t implicitly claim credit for another artist’s work just for the sake of keeping a joke alive.

Since posting the image, I’ve tested other implementations of the same technique. Gogh looks a little better but he’s not the same. It will take more work with a network trained specifically on portraits to have even the slightest chance of reconstruction. But there’s still plenty to explore. And there’s always the next thing.

Image for post
Image for post
Walking through an abstract landscape.

Written by

Artist working with code.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store