Adobe has announced a bunch of new tools and features across Creative Cloud apps at Adobe MAX 2024, but one of the biggest pieces of news is that for the first time, users can use Firefly AI to generate video from text prompts. And they can do so directly in Premiere Pro, Adobe’s industry-standard video editing software.
While Adobe has been teasing AI video generation for some time – it first announced that features would be coming to Premiere Pro back in April – the first addition in Premiere Pro beta is still surprising video editors. While we’re not getting a full batch of generative AI video features yet, Generative Extend has huge potential for regular video editors, providing the ability to lengthen a clip to fill gaps in a timeline.
Now Live: Generative Extend in Premiere Pro (beta) | Adobe Video – YouTube
Like Generative Expand in Photoshop, the name of the new AI tool in Premiere Pro is fairly self-explanatory. It uses AI to extend clips or generate missing shots to fill short spaces on a timeline for the perfectly timed edit. It’s a solution to a fairly common problem in video editing: finding that a clip ends half a second too early, which often requires a clip to be slowed down or for a still to be used to fill the space. And it’s easy to access in the toolbar, making it as easy to use as adjusting an edit point.
“This is going to be so useful,” wrote, Florida Keys Film Commissioner Chad Newman on X. “I can’t tell you the number of times I’ve wished for just a few more frames. No more having to slow down the last 10 frames to make it fit game”. “Getting closer to my wish of just working in premier and not having to jump around to different platforms to generate,” the videomaker Allen T wrote.
There are limitations. For now, Premiere Pro’s AI video extension is limited to 1920×1080 or 1280×720 resolutions in a 16:9 aspect ratio at 12-30fps. Videos must be at least 2 minutes long. And while the tool can extend the room tone and sound effects too, it can’t extend music due to copyright.
Based on initial experiments, it doesn’t seem to work quite as smoothly as shown in Adobe’s demo video, but the speed of generation is surprisingly fast. It sometimes struggles with a lot of motion, but it can work well on more static shots. It can also be used to add additional media for J or L cuts to to correct eye-lines or actions that move mid-shot. And this is only the beta version. It’s very promising for the future of Premiere Pro. When Adobe gets round to adding an AI object remover as well, the program is all but guaranteed to keep its spot at the top of our guide to the best video editing software.
Text-to-Video and Image-to-Video in Adobe Firefly
(Image credit: Adobe)
Adobe has also announced that it’s launching standalone access to the Firefly AI video model online through the Firefly site. This implementation will allow text-to-video prompting, including the ability to use a variety of camera controls. And users will also be able to generate video from stills using reference images, similar to what’s currently offered by tools such as Runway.
Users will be able to use a reference image alongside a text prompt to, for example, create a complementary shot for existing content, such as a close-up, by uploading a single frame, or to create new b-roll from still photography.
The Firefly video model will also be able to create atmospheric elements like fire, water, light leaks, and smoke on a black or green background so they can be layered over existing video using blend modes or keying in Premiere Pro or After Effects. Another potential use is to visualise creative intent for difficult-to-capture or expensive shots, allowing quick turn arounds before going into VFX or back to set for pick-up shots, and removing the need for the dreaded ‘insert shot here’ placeholder.
Much like with the original Adobe Firefly AI Image Model, Adobe is billing Firefly Video as the first publicly available AI video model designed to be safe for commercial use. What it means by that is that the model was trained on licensed content, which it believes will prevent users from facing any potential legal challenges.
The standalone Firefly AI video model is currently available only through a limited public beta open to a “small group of creative professionals”. Image-to-Video clips are limited to five seconds, and the quality to 720p and 24 frames per second. But, based on initial demonstrations, it looks like it provides good prompt adherence to detailed prompts and impressive clarity. And the addition of a Firefly video model to join the existing Image Model, Vector Model and Design Model, looks set to make Firefly the most complete generative AI model for creatives.
Adobe’s latest update comes as competition in AI video hots up following the unveiling of Meta MovieGen, but, unlike Adobe, Meta does not plan to open public access to its AI video model any time soon. Access to Adobe Premiere Pro and other Adobe apps is available via a subscription to Creative Cloud. See the best prices in your region below.
