Reimagining Videos with AI: An Artist’s Experience with Deforum and Runway ML’s Gen-1
As an artist spending time at the Catwalk Artist Residency, I decided to take on a unique challenge: harnessing the power of artificial intelligence (AI) to restyle and reimagine the videos I had captured during my stay. I chose two distinct platforms for this journey: Deforum, utilizing Stable Diffusion for image generation, and Runway ML’s Gen-1 video-to-video platform. This is a tale of my experience with both platforms and the surprising outcomes they produced.
Deforum and Stable Diffusion: A Robust but Challenging Tool
The first platform I dived into was Deforum. Since it uses Stable Diffusion for image generation, it is a capable tool for video reimagination. An advantage is that it’s open source and free, making it accessible to everyone. However, the platform has its limitations.
For starters, Deforum’s use currently necessitates either coding skills or a public Google Notebook. This may prove a daunting barrier for those without a programming background. Similarly, the user interface and settings require some understanding of technical jargon, which can pose challenges to newcomers.
In terms of efficiency, Deforum is a bit of a slow burner. Generating the frames and converting them to an animated movie file takes a considerable amount of time. This can be a significant setback for users wanting quicker results.
Another noticeable issue was the quality of output. The transition from one frame to another was often choppy, though some manual tinkering with the settings did smooth things out to an extent. Despite these limitations, Deforum offers a robust tool for those willing to invest time and effort into learning its intricacies.
Runway ML’s Gen-1: A Simple, User-Friendly Alternative
As I moved on to Runway ML’s Gen-1, the difference was immediately noticeable. While you can try it for free, there are limitations. Free users can only produce three video outputs, each no longer than five seconds. For longer videos (currently up to 15 seconds) and unlimited outputs, a $12 monthly fee applies.
But what sets Gen-1 apart is its user-friendly interface. There’s no need for coding skills; the settings are simple to navigate, and the customization options are vast. Users can upload a video and restyle it based on three methods: uploading an image, selecting from a list of preset styles like claymation, illustration, and futuristic, or inputting a text prompt like “fish swimming in a river”.
An incredibly useful feature of Gen-1 is its preview function. Before committing to full video generation, you can generate a still example of what the style will look like. This significantly helps in the decision-making process, allowing users to choose the stylistic option that best aligns with their vision before fully generating the video.
What truly stands out is the flexibility Gen-1 provides in terms of restyling strength. A single slider allows you to adjust the effect to your liking, or, for more refined control, you can switch to the advanced mode.
The quality of Gen-1’s outputs was superior to that of Deforum’s. Transitions were smoother, the videos cleaner, and overall, the process was less time-consuming. This lower barrier to entry, combined with the option to preview styles, makes Gen-1 a great option for artists venturing into AI-based video restyling.
Final Thoughts
My experience with Deforum and Runway ML’s Gen-1 was an enlightening journey, exposing the potential of AI in the realm of artistic expression. While Deforum presents a robust tool with vast possibilities for those with a programming background, Runway ML’s Gen-1 offers a more user-friendly alternative for those seeking simpler, quicker results.
AI’s role in the art world is only just beginning. As artists, we stand on the precipice of a brave new world, ready to push boundaries and create in ways we never thought possible.
