01WhatMonet, in Van Gogh's hand.
Moving the style of one image onto another is useful well beyond art — for photos and film, too. We used generative adversarial networks to learn a painter's style and apply it to new images: Monet to Van Gogh, and Van Gogh to Monet. The data came from Kaggle competitions: 300 Monet paintings, 540 Van Gogh paintings and 7,028 photos, all 256 × 256.
02HowCycleGAN with U-Net generators.
There are no paired examples of the same scene painted by both artists, so we built on CycleGAN: two generators translate between the domains and back, two discriminators tell real from generated, and a cycle-consistency loss makes sure an image can make the round trip. Each generator is a U-Net — seven convolution layers (kernel 4) down, seven transpose convolutions up, with skip connections at each depth. The discriminator is kept deliberately simple, because a discriminator that is too strong drags the whole model down. Loss: binary cross-entropy; optimizer: Adam.
03FailuresWhat didn't work.
- Translating Monet ↔ Van Gogh directly, with only a few hundred paintings, barely did anything — mostly a swap of blue and yellow.
- A shallow network (no GPUs to spare) underfit and produced low-quality images.
- A large learning rate (0.002) let the discriminator become almost perfect — its loss near zero — and the generator failed.
04The fixTake the detour through photos.
We trained two CycleGANs instead: Monet ↔ photo and Van Gogh ↔ photo. To repaint a Monet as a Van Gogh, turn it into a photo first, then into a Van Gogh. It worked — the new model changed not only the hue but the way of painting.
05NextWhere it could go.
- Add more portraits — Van Gogh painted many, and they transferred poorly.
- Train the two CycleGANs together instead of separately.
- Compare with StyleGAN, and combine the two.
Adapted from my original write-up · more in the links above.