Nov 19, 2020 6 min read

Add A New Dimension To Your Photos Using Python

Breathe New Life Into Your Photos With A 3D Ken Burns Effect

An Exploration Into Papers With Code

Some Backstory

After looking through some photos from a recent camping trip that I had taken a couple of weeks ago. I decided to take some of my old vacation photos, and turn them into a slideshow with some Python. Don’t worry I undertook this endeavor with the understanding that Google Photos, and the Apple equivalent do this really well already. This was really just intended to feed my addiction to building out side projects.

When looking through the available options I stumbled across a paper with some code that would take your photos, and add a transition that applies a 3D Ken Burns Effect to your photos. The Ken Burns Effect gets the 3D aspect for free with video obviously, but is less compelling with photos due to that missing dimension. So in the following write up I am hoping to set you up to be able to try, and add this extra dimension to your photos as I was able to.

Concepts

Here are some concepts that are worth knowing due to their relevance in what we are looking to achieve. Though we don’t need to be aware of this the AI models are applying the 3D dimension to our Ken Burns effect by adding perspective through correctly scaling the size of the objects in the photo through estimating their depth. As the effect moves through the photo the model is required to reconstruct the missing objects, and textures that don’t appear in the 2D image through inpainting.

Ken Burns Effect — A zooming and panning across photographs gives the feeling of motion. It’s named after an American documentarian who often used this effect. The writers of this paper have added the third dimension to this that would normally be missing from photos.

Depth Estimation — Describes the method for obtaining a measure of the distance of the objects in a scene. Doing this effectively will result in a convincing transition that will scale each object with the parallax.

Inpainting — This describes the method for reconstructing the part of the image that doesn’t exist due to the changing perspective of this transition. In images, and videos this technique also refers to repairing damaged, deteriorating, or missing parts of an artwork are filled in to present a complete image.

DIY

Luckily the authors of this paper, Simon Niklaus, Long Mai, Jimei Yang, Feng Liu, have provided us with a codebase that allows us to apply this effect to our photos with some pre-trained models. I have noticed that they don’t work well for photos with lots of sharp edges in the foreground, but yield pretty impressive results on a number of the photos that I have tested it with.

I won’t get into the details into how this is achieved as the authors do this in their paper, Github Repo, and Youtube video which all have been provided at the end in the resources section.

I am going to step you through the approach that we take to create these 3D scenes, but if you want to just blindly try it yourself at the end of this section I added all of the code that we will be going through to a Google Colab that you can copy, and use immediately.

Before stepping through this yourself I suggest opening a new Google Colab, because in the code samples I go through I use some Colab specific functionality to help with ease of use.

Setup Environment and Install Requirements

The first step to get this working yourself is to import their repository that contains this implementation, and pre-tained models. So what happens below is we clone the repo, navigate to it, make our output directory, and install the required libraries. The Google Colab has a lot of the requirements already preloaded when you start a new runtime so there was just two that were missing.

!git clone https://github.com/sniklaus/3d-ken-burns.git

# Move into the downloaded repository
%cd 3d-ken-burns

# Make a directory for the videos
!mkdir videos
video_dir = "./videos/"

# Install dependencies
!pip install moviepy gevent

Upload Photos

At this stage all that you need to do is have an easy way to upload photos to your source/image directory. You don’t really need to include this in your notebook to move forward, as you could just drag your files into their corresponding folders with the Colab interface, but it’s a pretty nice connivence cell.

from google.colab import files

uploads = files.upload()

for filename in uploads.keys():
    !mv ./$filename ./images/$filename

Convert Your Photos

So if we didn’t want to convert all of our photos that we provide to the two directories, and easy one liner would be as follows which just uses the autozoom.py interface to convert one photo.

python autozoom.py --in {input_image}.jpg --out {output_video}.mp4

Though since we want to automate what we can, we can take advantage of bash, and iterate through the directory then execute on each photo.

!for image in ./images/*; do python autozoom.py --in $image --out ./videos/$(basename $image | cut -f1 -d '.').mp4; done

View Your Video

Once your photos are converted you may want to view the output first we use IPython’s functionality to build the function to view the video, and then provide a selector for us to choose which video to view amongst those in the directory.

import os
from base64 import b64encode

from IPython.display import HTML
import ipywidgets as widgets

def video(path):
    mp4 = open(path,'rb').read()
    data_url = "data:video/mp4;base64," + b64encode(mp4).decode()
    return HTML(f'<video width=600 controls loop> <source src="{data_url}" type="video/mp4"></video>')

files_list = os.listdir(video_dir)
video_list = widgets.Dropdown(
    options=files_list,
    value="" if not files_list else files_list[0],
    description='Video:',
    disabled=False,
)
display(video_list)

Convert Mp4 to Animated GIF

I added this step as a result of wanting to upload, and embed the output of this applied effect in this post, so as with all of my efforts, please take advantage of this knowledge, and convert your mp4’s to animated gif’s as you please. As a bonus we have already imported imageio as a dependency so there’s no need to install it. Below we use this function, and iterate through the destination directory looking for mp4’s to convert.

import imageio
import os, sys

class TargetFormat(object):
    GIF = ".gif"
    MP4 = ".mp4"
    AVI = ".avi"

def convertFile(inputpath, targetFormat):
    """Reference: https://gist.github.com/michaelosthege/cd3e0c3c556b70a79deba6855deb2cc8"""
    outputpath = os.path.splitext(inputpath)[0] + targetFormat
    print("converting\r\n\t{0}\r\nto\r\n\t{1}".format(inputpath, outputpath))

reader = imageio.get_reader(inputpath)
    fps = reader.get_meta_data()['fps']

writer = imageio.get_writer(outputpath, fps=fps)
    for i,im in enumerate(reader):
        sys.stdout.write("\rframe {0}".format(i))
        sys.stdout.flush()
        writer.append_data(im)
    print("\r\nFinalizing...")
    writer.close()
    print("Done.")

for file in [x for x in os.listdir(video_dir) if x.endswith(".mp4")]:
    convertFile(f"{video_dir}{file}", TargetFormat.GIF)

Download All Videos

If you don’t want to use the file explorer to download your output. The last step is to iterate through the destination folder, and download each video. This could also be done as we did previously when using bash to iterate, and apply the effect.

for file in os.listdir(video_dir):
    files.download(f"{video_dir}{file}")

If all has gone well then your photos have been properly converted as one of mine I took here when enjoying Bolonga when times we different.

For those that want to skip putting a Colab notebook together themselves here is the complete implementation for you to make a copy of, and start converting your photo albums.

Google Colaboratory
Edit descriptioncolab.research.google.com

Key Takeaways

Standing on the shoulders of giants we live in an age where we can take advantage of cutting edge AI techniques to help us do things that normally would require someone with specific knowledge. In this instance we can use the AI techniques developed by Simon Niklaus and team instead of needing to learn sophisticated editing skills.

Resources

Video Description
Research Paper
Github Repo
Gist for MP4 To GIF Conversion
Other Google Colabs Leveraging Paper from Arnaldo Gabriel, one from Vlad Alex, and one from Ahmed Harmouche.

Enjoying these posts? Subscribe for more

Subscribe now