Another player has emerged in the race to create a reliable AI video generator – none other than Netflix. Researchers from the streaming giant were involved in the development of Go-with-the-Flow, a new model that aims to provide an easy way to control motion patterns in video diffusion models.
Appearing hot on the heels of Tencent’s HunyuanVideo, the model lets users decide how objects in a scene will move simply by cutting and dragging. And it allows camera control and even motion transfer from one video to another.
Go-with-the-Flow: Motion Controllable Video Diffusion using Real-Time Warped Noise – YouTube
Watch On
Go-with-the-Flow was developed by researchers from Netflix, Netflix Eyeline Studios, Stony Brook University, University of Maryland and Stanford University. It allows users to animate an object simply by selecting an object and defining a trajectory by dragging like you might do with a mask in video editing software.
First, a crude animation is created as a cut out, then warped noise is added in a diffusion script (this part requires GPU) to turn this into smooth(ish) animation. Like most AI video, the results look janky for now, particularly with people and animals since limbs sometimes get mixed up and appear to trip over each other, but it’s still a notable advance.
The researchers say Go-with-the-Flow simply fine-tunes a base model, requiring no changes to the original pipeline or architecture, except the use of warped noise instead of pure IID Gaussian noise. The model also allows motion transfer, using the original video as the motion signal and a new target prompt to guide the output. And it can create turntable animations using a 3D-rendered turntable camera motion as a guide.
The code is available at GitHub and Hugging Face. You can see a tutorial on the AI video model below.
Go-with-the-Flow Cut-and-Drag Tutorial – YouTube
Watch On
Netflix’s involvement in the project is intriguing. The streaming platform has already announced that it’s working on using generative AI to make games, but this project adds weight to suspicions that it may be working on AI for video generation too.
That could take the form of something like what Lionsgate, the company behind John Wick, is doing with Runway, or more radically, perhaps Netflix will explore something like the approach of Showrunner AI, which thinks users will want to use AI to make their own shows. For now, the company’s occasional use of AI art has already earned it a lot of criticism.
