How to use OpenPose and ControlNet in Stable Diffusion

In the realm of character animation, the ability to transfer poses seamlessly between different characters is a valuable asset. Traditionally, achieving this involved complex 3D software and time-consuming processes. However, with the advent of OpenPose and its integration with Stable Diffusion, a revolutionary solution has emerged. In this article, we delve into the remarkable capabilities of OpenPose and how it synergizes with Stable Diffusion, opening up new possibilities for character animation.

To transfer the pose of a character using ControlNet and the Open Pose Editor extension in Stable Diffusion, follow these steps:

Step 1: Install ControlNet Extension

• Launch Stable Diffusion
• Click on Extensions in the menu.
• Select Install from URL.
• Paste Controlnet Github URL: https://github.com/Mikubill/sd-webui-controlnet
• Click on Install to add the extension to your workspace.

• To ensure successful installation, go to the Installed section and check for updates.
• After confirming the installation, click on Apply and restart UI to apply the changes and restart.

Step 2: Install OpenPose Editor Extension

• Launch Stable Diffusion
• Click on Extensions in the menu.
• Select Install from URL.
• Paste OpenPose Github URL: https://github.com/fkunn1326/openpose-editor
• Click on Install to add the extension to your Stable Diffusion UI.

• To ensure successful installation, go to the Installed section and check for updates.
• After confirming the installation, click on Apply and restart UI to apply the changes and restart.

Prior to utilizing the blend of OpenPose and ControlNet, it is necessary to set up the ControlNet Models, specifically focusing on the OpenPose model installation. For this you can follow the steps below:

• Go to ControlNet Models
• Download all ControlNet model files (filenames ending with .pth)

Or just download the OpenPose model "control_v11p_sd15_openpose.pth"

• Put the model file(s) in the ControlNet extension’s models directory:

Now that you have the OpenPose Model & Open Pose Editor Extension installed, you can easily edit and pose stick figures. Within the OpenPose Editor tab, you'll find a dedicated box where all the manual editing takes place. The extension mimics the shapes and colors of the Open Pose model, ensuring consistency and ease of use. Adjust the stick figure's pose as desired within the designated box.

Step 1: Create Pose & Send Pose to ControlNet

With the newfound capability, you can now click on the button Send to txt2img below. By clicking this button the stick figure pose will be seamlessly transmitted to ControlNet.

Step 2: Enable ControlNet Settings

• To enable ControlNet, simply check the checkboxes for Enable and Pixel Perfect (If you have 4GB of VRAM you can also check the Low VRAM checkbox).
• Select OpenPose as the Control Type
• Select None as the Preprocessor (This is because the image has already been processed by the OpenPose Editor).
• Select Control_v11p_sd15_openpose as the Model

Step 3: Create Prompt and Generate Image

To create stunning images, you must select the checkpoint and carefully craft a prompt for the character. This allows you to bring your artistic vision to life. With everything set, a single click on the Generate button unleashes the transformative power of ControlNet and the OpenPose Editor, turning your scene into a dynamic masterpiece.

• Positive Prompt: ((detailed)), ((masterpiece)), ((best quality)), ((Astounding)), ((Enchanting)), (Striking), woman, (lara croft:1.2), (tomb raider:1.2), (tomb:1.1), natural light inside tomb, beach, beach background, sunny, jungle, plants in background, beach background, beach, tropical beach, water, (detailed face:1.2), clear skin, (cleavage:1.1), perfect light, perfect shadows
• Negative Prompt: (worst quality:1.2), (low quality:1.2), (lowres:1.1), (monochrome:1.1), (greyscale), multiple views, comic, sketch, (((bad anatomy))), (((deformed))), (((disfigured))), watermark, multiple_views, mutation hands, mutation fingers, extra fingers, missing fingers, watermark
• Sampling Steps: 30
• Sampler: DDIM
• CFG scale: 7
• Size: 912x512 (16:9 ratio)
• Model: reV Animated

Presenting the Dynamic Poses Package, a collection of poses meticulously crafted for seamless integration with both ControlNet and the OpenPose Editor. This package offers an array of expressive poses that can effortlessly be employed, enhancing the visual appeal and interactivity of your projects.

Included in Downloads

The following is included when you download the poses:

• Preview of Pose
• JSON file of Pose
• Stick figure of Pose

Additionally there is a presets.json file with all poses in JSON format.

How to use?

To use with ControlNet & OpenPose:

• Drag & Drop the Stick Figure Poses into a ControlNet Unit and do the following:
• To enable ControlNet, simply check the checkboxes for Enable along with Pixel Perfect.
• Select OpenPose as the Control Type
• Select None as the Preprocessor (Since the stick figure poses are already processed)
• Select Control_v11p_sd15_openpose as the Model

To use with OpenPose Editor:

For this purpose I created the presets.json file, which can be found in the downloaded zip file. You can place this file in the root directory of the openpose-editor folder within the extensions directory:

The OpenPose Editor Extension will load all of the Dynamic Pose Presets from the presets.json file. You can then choose a pose from the Presets dropdown menu, and by clicking Load Preset, you'll be able to preview the selected pose. From there you can send it to either "txt2img" or "img2img".

If you are still unsure how all of this works, please checkout the video above.

If you wish to have multiple characters in the same scene, the Open Pose Editor extension has you covered. Simply click on the Add button to add a new skeleton. Once added, you can pose and adjust it to your liking. Repeat this process to introduce additional characters into the scene, each with their unique poses. If you are happy with the poses you can again send it to ControlNet via the Send to txt2img button below the stick figure canvas.

Before clicking on Generate in the txt2img tool, be sure to steps below:

• To enable ControlNet, simply check the checkboxes for Enable and Pixel Perfect (If you have 4GB of VRAM you can also check the Low VRAM checkbox).
• Select OpenPose as the Control Type
• Select None as the Preprocessor (This is because the image has already been processed by the OpenPose Editor).
• Select Control_v11p_sd15_openpose as the Model

And of course write a good prompt! Once done, simply wait for the magic to unfold as the generated image appears before your eyes.

Examples of Multiple Characters in a Scene

• Positive Prompt: ((detailed)), ((masterpiece)), ((best quality)), ((Astounding)), ((Enchanting)), (Striking), woman, lara croft twin, twin women, twins, (lara croft:1.2), (tomb raider:1.2), (tomb:1.1), natural light inside tomb, beach, beach background, sunny, jungle, plants in background, beach background, beach, tropical beach, water, (detailed face:1.2), clear skin, (cleavage:1.1), perfect light, perfect shadows
• Negative Prompt: (worst quality:1.2), (low quality:1.2), (lowres:1.1), (monochrome:1.1), (greyscale), multiple views, comic, sketch, (((bad anatomy))), (((deformed))), (((disfigured))), watermark, multiple_views, mutation hands, mutation fingers, extra fingers, missing fingers, watermark
• Sampling Steps: 30
• Sampler: DDIM
• CFG scale: 7
• Size: 912x512 (16:9 ratio)
• Model: reV Animated

In conclusion, utilizing Open Pose and Control Net together within Stable Diffusion unlocks a world of creative possibilities for digital artists. The efficient integration of these tools enables users to detect and manipulate poses with ease, producing stunning images that accurately reflect artistic intentions. This crash course has equipped you with the essential knowledge to harness the power of pose detection and streamline your artistic workflow. By consistently refining your approach and exploring the innovative features available, you can elevate your digital artistry to remarkable heights. Thank you for engaging with this guide; may your creative journey be filled with endless inspiration and captivating visuals.