While there are doubts about its ethics, Midjourney remains one of the best AI image generators in terms of prompt interpretation and photorealism. But with the launch of Midjourney Patchwork, it wants to be much more than that – a complete creative software package for ideation, storyboarding and more.
Rather than merely allowing the generation of AI images, the aim is to allow users to create entire worlds, with coherent aesthetics and stories. And the company even has plans to expand it into the 3D and VR spaces.
Today we’re opening up a new experimental multiplayer worldbuilding tool called “Patchwork”. It combines language models, image models, and a canvas-based interface to build out the foundations of stories. Check out the 🧵below for links and further documentation. Have fun! pic.twitter.com/IgoJrJ8bO9December 11, 2024
What is Midjourney Patchwork?
Midjourney Patchwork uses an infinite canvas, which will be a familiar concept for anyone who’s seen Adobe Project Concept as well as Recraft AI image generator and Kaiber Superstudio. We’re told that it uses three different large language models wrapped up in the form of a standalone web app that requires a Midjourney account for access.
There’s a ‘Toolbox’ on the left of the canvas with a variety of options to add entities to the canvas, for example character, event, place and prop. Each canvas is called a ‘world’. Elements called ‘scraps’, which include images and text boxes, can be dragged around the canvas freely, and users can switch between worlds by creating a ‘portal’ – a small circular button.
To generate a new world, you enter a text prompt in an editor bar at the top of the ‘Create’ screen and select from a set of styles. By adding new character boxes, the user can prompt names, characteristics and motivations to develop a story. Characters can then be linked with lines that mark the connections between them.
Midjourney Patchwork also adds collaboration options since users can share a board with others who, we’re told, will be able to work on it at the same time, take a nod from Figma’s approach. Max Kreminski, leader of Midjourney’s Storytelling Lab, said in a demo on Discord that a world can support up to 100 users, although the more users, the more chaotic it could become.
One thing that stands out in the demo above is that Midjourney is basically doing all the work. We’re constantly told that AI is merely a tool for human creativity, but, in this particular demo at least, Midjourney isn’t just generating the images, it’s basically proposing the entire story, with the human’s role reduced to that of deciding what elements to use and which to reject, almost like the reader of a choose your own adventure book.
It seems that users can opt to take a more active role in the process if desired. Midjourney has said that fairly soon users will be able to upload multiple reference images to boards to guide generations. For now, Midjourney Patchwork is available to Midjourney users at https://patchwork.midjourney.com. Several people have already made tutorials on how it works (see the video from Future Tech Pilot below).
What’s coming next to Midjourney?
Beyond the tech itself, the launch of Midjourney Patchwork is interesting from a strategic point of view. It suggests that Midjourney is out to compete with more traditional and practical creative software. The company plans to launch Midjourney V7 soon introducing multiple character consistency across images, and it’s revealed more ambitious plans for the future.
Kreminski said during the demo of Midjourney Patchwork that the company has a “very clear path in terms of escalation of the details and interactions in the worlds,” and that it plans to eventually allow AI generation of immersive 3D virtual reality scenes. That will presumably be several years away.
Meanwhile, Midjourney Founder David Holz has said that the company is working hardware projects and that it aimed to “branch out and become a full research lab”.
For all the advances, AI art remains controversial and continues to generate a public backlashes when used by major brands. Just see the reaction to uses as diverse as the Coca-Cola AI Christmas ad and Call of Duty Black Ops 6.