Create 3D Zoom Animations with Stable Diffusion (Depth Map)

Welcome to the world of cutting-edge 3D Zoom Animation, where the power of Stable Diffusion meets the versatility of Depth Map Extension. In this exciting journey, we'll delve into the art of creating mesmerizing animations that bring your static images to life with stunning detail and depth. Join us as we unlock the secrets of immersive visuals and dynamic transformations, opening up a new dimension of creative possibilities, lets get in the groove!

Before we can make a still image come to life with a 3D zoom animation, we should ensure that certain prerequisites are met.

Requirement 1: Depth Map Extension

This software serves as an extension for AUTOMATIC1111's Stable Diffusion WebUI, designed to generate depth maps. It has the capability to produce 3D stereo image pairs (side-by-side or anaglyph), normal maps, and 3D meshes using either automatically generated or custom depth maps. For further details, you can visit the official Depth Map GitHub Page.

Let's get started with the process of integrating the Depth Map Extension into your Stable Diffusion interface, enhancing its capabilities and enabling the creation of stunning 3D effects. To guide you through this installation, we've outlined all the necessary steps for your convenience, which you can find listed below.

• Go to the 'Extensions' tab.
• Click on 'Available.'
• Select 'Load from.'
• Search for the 'Depth Maps' extension.
• Click "Install"

After successfully installing the depth maps extension, head over to the "Installed" tab, and choose "Apply and restart UI." I strongly advise conducting a full restart of your Stable Diffusion for a smooth experience and to avoid any possible errors.

If the installation of the depth map extension was successful, you should notice a new addition to the top navbar within Stable Diffusion labeled as "Depth".

Requirement 2: Initial Image

Prepare an initial image that will be used to create a 3D zoom animation effect with the depth map extension. For this tutorial, we'll kick things off with an image created using the 'txt2img' tab in Stable Diffusion. You can see the initial image we'll be using right below, 9:16 ratio:

Initially, we'll generate a mesh file, which will be used later in the process to craft the 3D zoom animation. A mesh file is a digital file that encodes the geometric structure of a 3D object/image.

To complete this task, it is vital to adhere to the provided instructions, as the settings in the Depth Map extension are essential for mesh file creation.

• Click on the "Depth" tab in the top navbar.
• Proceed to upload the initial image to the canvas within 'Single Image' sub-tab
• Compute on: GPU (Select 'CPU' if your VRAM is less than 8 GB).
• Model: res101
• Enable the "BOOST" checkbox ("Enabling 'BOOST' will activate multi-resolution merging, leading to substantial enhancements in the quality of results. For optimal outcomes, it is recommended to use the 'res101' model).
• Enable "Save Outputs"
• Enable "Generate 3D inpainted mesh" (I recommend not selecting the option to generate four demo videos. In my experience, these demos often exhibit inaccuracies, especially in the X dimension, which can result in videos with blurred and indistinct edges).
• Click the 'Generate' button

Once the rendering process is finished, you will find a successfully generated mesh file in the directory path: 'stable-diffusion-webui' > 'outputs' > 'extras-images'.

Once we have acquired the mesh (.obj) file, we can continue by navigating to the right side of the Depth extension interface, where we will select the "Generate video" sub-tab.

Next, let's adjust the following parameters to shape our 3D zoom animation with the mesh file we previously produced. Of course, you're welcome to explore and experiment with your own creative twists in the three-dimensional space.

• Input mesh: outputs/extras-images\depthmap-0000.obj (Change the name of the .obj file as necessary)
• Number of frames: 240
• Framerate: 24
• Format: mp4 (Choose 'webm' for superior compression, perfect for enhancing web performance.)
• SSAA: 4 (Supersampling Anti-Aliasing (SSAA) can be used to get rid of jagged edges and flickering. The render size is scaled by this factor and then downsampled.)
• Trajectory: double-straight-line
• Translate: x, y, z: -0.005, 0.005, -0.05 (Don't hesitate to explore these settings and experiment to achieve your desired results).
• Crop: top, left, bottom, right: 0.03, 0.03, 0.05, 0.03 (Default)
• Uncheck the "Dolly" Checkbox (Dolly option adjusts the FOV so the center subject will stay approximately the same size, when enabled)
• Click "Generate"

Note: Keep in mind that input mesh file path location is relative (outputs/extras-images\depthmap-0000.obj).

After the rendering process wraps up, we can observe our 3D Zoom Animation in full motion. Below, we have a selection of examples featuring our stealthy ninja. You'll discover the successfully generated 3D Zoom Animation stored in the following directory path: 'stable-diffusion-webui' > 'outputs' > 'extras-images'.

In conclusion, our journey through the world of 3D zoom animation, powered by Stable Diffusion and its Depth Map Extension, has unveiled a realm of creative possibilities. From installing the extension to fine-tuning settings and generating mesmerizing animations, we've explored the steps necessary to breathe life into static images.

By mastering these techniques, you can create captivating animations that transcend the ordinary, giving your content depth and dimension. Whether it's for artistic expression, visual storytelling, or any other creative endeavor, the tools and knowledge gained here open up new horizons in the world of digital media.

Now, armed with the skills to transform images into dynamic, immersive experiences, you're ready to embark on your own creative journey, exploring the endless possibilities that 3D zoom animation has to offer. So, go forth and bring your visions to life with this exciting blend of technology and artistry.