How to make Amazing AI Videos with Deforum (Stable Diffusion)

Welcome to the fascinating realm of AI animation, where you can bring your ideas to life with stunning visual effects. In this step-by-step guide, we will walk you through the process of creating captivating animation videos using Stable Diffusion and the Deforum extension, the revolutionary AI-powered tool. Are you ready to embark on this exciting journey? Let's dive in!

To begin creating stunning AI animation videos, you'll need to install Stable Diffusion. If you haven't done this already, follow the instructions in the following tutorial:

How to Install Stable Diffusion on Windows

Once you have set up and launched Stable Diffusion, you'll need to install an extension called Deforum. Here's how you can do it:

• Go to the extensions tab in Stable Diffusion.
• Under the "Available" section, click on "Load From" and search for "Deforum".
• Click on the Install button and wait for a few minutes for the installation to complete.

After installation, go to the "Installed" tab, click "Check for updates" after it’s done click on “Apply and Restart UI”. This will enable the Deforum extension tab to appear.

To achieve a specific visual style, you'll need to download a specialized AI model. You can learn more about Stable Diffusion checkpoints/models in our Model Installation Guide. In this tutorial, we'll use a model called reV Animated.

After downloading your specific model, you will have a checkpoint file in your downloads folder. To use the model, you need to move this file to the designated models folder in your Stable Diffusion installation directory. This process ensures that Stable Diffusion recognizes and utilizes the installed model during image generation. Make sure you place the downloaded stable diffusion model/checkpoint in the following folder "stable-diffusion-webui\models\Stable-diffusion"

To ensure the proper output, we need to adjust some settings in the Deforum extension. Let's go through each setting and its purpose:

If you want to skip these steps you can also download and import my settings.

• Simply download my settings file (right click on the link and click "save link as") and copy the path (Shift + Right Click > Copy as path).
• Paste the path in the settings file.
• Paste the path in the settings file.
• Press on “Load All Settings”. These settings will give you a 10 second video with a 720 x 1080 resolution (9:16) which is most commonly used for Social Media i.e. TikTok, Reels & Shorts.

Run

Under the "Run" tab:

• change the sampler to "DPM++ 2M SDE Karas."
• Set the steps to 25. You can increase this value to improve animation quality.
• Set the dimensions to 720 by 1280. You can upscale it to a higher resolution later if desired.
• Keep the Batch name as default; “Deforum_{timestring}”. Keep the remaining settings at their default values.

Keyframes

In the “Keyframes" tab:

• Set the animation mode to "3D".
• Change the border to "Wrap."
• Set the number of frames to determine the length of your animation. For this tutorial, we'll use 150 frames.
• Set the strength schedule to “0: (0.65),25: (0.55)”, which starts with a strength of 0.65 and changes to 0.55 at frame 25. Keep the remaining settings at their default values.

Motion

In this sub tab we can adjust the camera movements and rotation, each parameter explains what it does by hovering your mouse over them.

You can copy the settings I'm using or get creative with your own variations.

• Translation X: 0:(0), 8:(15), 52:(15), 75:(0)
• Translation Y: 0:(0)
• Translation Z: 0:(0.2), 15:(10), 75:(15)
• Translation 3D X: 0:(0), 15:(0), 22:(0.5), 45:(0.5), 75:(0.5)
• Translation 3D Y: 0:(0), 8:(-3.5), 22:(-2.5), 45:(-2.8), 75:(-2), 105:(0)
• Translation 3D Z: 0:(0), 15:(0.2), 22:(0), 45:(-0.5), 75:(0), 105:(0.5), 125:(0.8) Keep the remaining settings at their default values.

Noise

• Noise type: perlin.
• Noise schedule: 0:(-0.06*(cos(3.141*t/15)**100)+0.06) Leave the rest of the settings unchanged, using their default values.

Anti Blur

• Amount schedule: 0: (0.05) Keep the remaining settings at their default values.

Depth Warping & FOV

• Important: If you have an AMD GPU uncheck the “Use depth warping” box.
• FOV schedule: 0:(120) Keep the remaining settings at their default values.

Setting up prompts allows you to control the progression and changes in your animation. Here's how: By default, we start with a specific prompt at frame 0. At frame 15, the AI will start generating with the new prompt. You can add your own prompts to personalize your animation. Including subject and style descriptions in your prompts can be helpful. Make sure the prompts are in the right format to prevent any errors.

{ "0": "unique prompt 1", "15": "unique prompt 2", "30": "unique prompt 3" }

You can also use ChatGPT to generate the prompt for you by copying the following text and changing a few parameters:

I have a photo of “DESCRIBE PHOTO HERE”. I want to animate this with ai to different versions of the photo. I have 150 frames in total and I earlier described frame 0. Give me creative prompts to change the frames 0 to 150. Give me prompts for frames 0-150, with steps of 15 frames in between each prompt. Give me specific creative prompts which are “realistic/futuristic/fantasy etc.”. Give me the prompt in the following JSON code format: { "0": "unique prompt 1", "15": "unique prompt 2", "30": "unique prompt 3" }

If your goal is to seamlessly transition from your own video to the AI animation, follow these steps:

• Enable "use init" to inform the AI that you want to start diffusing from the first frame using your own image.
• Use any video editing software (e.g., Premiere Pro) to extract the initial frame from your video.
• Copy the path of the exported file by right-clicking on it and selecting "Copy Path". On windows 10 use Shift + right click and then copy path.
• Paste the path into the "Init Image" setting in Stable Diffusion.

After your AI animation is done generating you can seamlessly stitch the 2 videos together with any video editing software.

Configure the output settings to finalize your AI animation

• Set the frame rate to 15 (adjust according to your preference). Leave the rest as default, you can always upscale your video later.

Finally, click on "Generate" to generate your the AI animation. The process may take some time based on your hardware and settings. Once it's complete, you can find the final output saved in your Stable Diffusion directory. Navigate to the "outputs" folder, then "image to image," to access your Deforum animations. Each animation folder will contain individual frames and a video file.

Congratulations! You have successfully installed Stable Diffusion, configured the Deforum extension, and created a stunning AI animation video. Feel free to experiment with different settings and prompts to unleash your creativity.

In conclusion, this tutorial has provided you with the necessary steps to install Stable Diffusion, install the Deforum extension, and create amazing AI animation videos. By following the instructions and exploring different settings and prompts, you can unlock your creative potential and produce captivating visuals. Remember to experiment, try out various styles, and make your animations unique. With the knowledge gained from this tutorial, you are well-equipped to embark on your journey of creating stunning AI animations. Have fun and enjoy the process of bringing your ideas to life with the power of AI!