How To Style Your Face Portrait with Stable Diffusion (ControlNet & IP-Adapter)

Welcome to the world of Face Portrait Styling! This guide is your passport to mastering the nuances of Stable Diffusion, ControlNet, and IP-Adapter technologies, transforming your face portraits into personalized works of art. Whether you want your photos to look really polished or just want to make your creative selfies stand out, this guide will make the whole process clear and easy. It combines innovation and style effortlessly, promising a straightforward journey into the world of face portrait styling. Let's dive in.

Before diving into the exciting realm of face portrait styling, let's ensure you have the essentials ready for this creative journey. Here are the key requirements to make the most of ControlNet and IP-Adapter technology:

ControlNet

To get started, it's essential to have the ControlNet Extension installed, which will serve as our backbone. Haven't installed the ControlNet Extension?

How to Install ControlNet Extension in Stable Diffusion (A1111)

IP-Adapter Face Model

To transfer and manipulate your facial features effectively, you'll need a dedicated IP-Adapter model specifically designed for faces. This model will be instrumental in accurately reflecting your facial expressions and features. For this you can visit the Hugging Face IP-Adapter Models page. Here you should download the following safetensors file:

Note: Please be aware that the safetensors file requires the latest ControlNet Version to function correctly. Ensure you "Check for updates" and restart the UI to update the ControlNet Extension to the most recent version.

File Location IP-Adapter

Once you downloaded the "ip-adapter-plus-face_sd15.safetensors" you need to do the following to make it ready for use:

• Place the file inside the following folder: "stable-diffusion-webui\extensions\sd-webui-controlnet\models"

Having these essentials in place sets the stage for a seamless and creative face portrait styling experience, unlocking the full potential of ControlNet and IP-Adapter technologies.

Now that the ControlNet extension and the IP Face Adapter Model are successfully installed, let's proceed to verify their visibility within the Stable Diffusion web UI. Navigate to either the "txt2img" or "img2img" tab, and as you scroll down, you should observe the expansion panel for the ControlNet extension.

ControlNet Extension & IP Face Adapter Model

Upon opening this panel and selecting the "IP-Adapter" Control Type, ensure that the "ip-adapter-plus-face_sd15" model is available for selection.

Troubleshooting

If, for any reason, you do not find this model in the options you can try two things:

• Click the blue refresh icon on the right side to update the models.
• If the model is still not visible, double-check the placement of the "ip-adapter-plus-face_sd15" file to ensure it is in the correct folder mentioned in section 2.

When everything looks right on the user interface, we can start creating our first styled face portrait. Let's give it a try!

In this section, we delve into the exciting process of creating a styled face portrait using the ControlNet canvas and the IP-Adapter technology. Let's do the following:

ControlNet Settings

• Begin by placing a Face Portrait of your preference onto the ControlNet Unit 0 canvas.
• Enable ControlNet by checking the "Enable" checkbox
• Enable "Pixel Perfect"
• Choose for the "IP-Adapter" as the "Control Type"
• Select the "ip-adapter-plus-face_sd15" model.
• Leave the rest of the ControlNet settings as the default

In our initial example, we'll feature the beloved Angelina Jolie as the stunning face model.

To enhance outcomes, create a prompt mirroring a traditional studio face portrait. In this initial attempt, we'll employ the ToonYou checkpoint for a charming cartoon-style effect.

Styled Face Portrait Examples

Face Portrait Prompt & Settings

• Positive Prompt: best quality, beautiful, 30 year old woman, (full face portrait:1.2), face portrait, detailed face, freckles, (facing camera), (looking into the camera), studio portrait, natural lightning, perfect shadows
• Negative Prompt: (worst quality:1.2), (low quality:1.2), (lowres:1.1), (monochrome:1.1), (greyscale), multiple views, comic, sketch, (((bad anatomy))), (((deformed))), (((disfigured))), mutation hands, mutation fingers, extra fingers, missing fingers, watermark
• Sampling Steps: 30
• Sampler: DPM++ 2M Karras
• CFG scale: 7
• Size: 512x512
• Model: ToonYou

Adjusting Prompt for Styling

Feel free to customize the prompt according to your preferences and make minor tweaks to the resulting face portrait. For instance, if I wish for Angelina Jolie to have red hair and blue eyes, I can include relevant keywords in the prompt to achieve the desired outcome, as demonstrated below:

• Adjusted Positive Prompt: best quality, beautiful, 30 year old woman, (red hair:1.2), (ginger hair:1.2), (blue eyes:1.2), (full face portrait:1.2), face portrait, detailed face, freckles, (facing camera), (looking into the camera), studio portrait, natural lightning, perfect shadows

Upscaling Styled Face Portrait (Hires. fix Settings)

If the outcome doesn't meet your expectations, you have the option to enhance your image using the "Hires. fix" feature. Typically, your selected checkpoint provides specific recommendations for upscaling settings. For the ToonYou Checkpoint, it suggests the following settings when upscaling your image:

• Upscaler: R-ESRGAN 4x+ Anime6B
• Hires Step: 14
• Denoising Strength: 0.35
• Upscale factor: 1.5+

Utilizing the upscaler enhances the image quality, providing superior results. Now that our initial styled face portrait with the ToonYou checkpoint is complete, feel free to experiment with various checkpoints for unique styles! Let's delve in and explore some of our favorite checkpoints using an XYZ plot.

In this section, we embark on a visual exploration by comparing various face portrait styles through the use of an XYZ plot. Haven't heard about the XYZ plotting technique within Stable Diffusion? Here's a simple guide:

XYZ Plotting to Compare Stable Diffusion Checkpoints

XYZ Plotting Script

This XYZ plotting method serves as a swift and efficient tool for assessing different checkpoints and their transformative impact on our initial face portrait. So let's do some plotting now. First click the X/Y/Z plot script under the "Script" expansion panel to open the options:

Once you have selected the X/Y/Z plot, a panel will appear where you can customize and fill in specific settings according to your desired comparison parameters. Now it is time for our comparison.

XYZ Settings & Comparison (Checkpoints vs Sampling Method)

Our comparison will be different checkpoints/models compared to different Sampling methods. To achieve this we will need the following settings:

• set X type to "Checkpoint name"
• at the X values select the checkpoints you want to compare
• set Y type to "Sampler"
• at the Y values select the sampling methods you want to compare

Below you can find our filled in settings for our XYZ plot.

You have the flexibility to include additional checkpoints and sampling methods. Moreover, there's the option to introduce another "Z type" to compare additional parameters. However, for the purpose of this tutorial, we'll keep it as outlined above.

Comparing Face Portrait Styles

Now, when crafting a new face portrait, you can quickly identify your preferred style at a glance. Personally, I'll stick to the ToonYou Checkpoint as I appreciate the cartoonish style it reflects.

Here are a few instances of styled face portraits created with the ToonYou Checkpoint, employing the settings and prompts detailed at the beginning of this article.

Jennifer Lawrence

Ana de Armas

Sofia Vergara

Emma Stone

In summary, we've delved into the realm of face portrait styling using the specialized IP-Adapter for faces within the ControlNet Extension. From setup to experimentation with various checkpoints, we've explored the creative possibilities at the intersection of technology and personal expression. The XYZ plot simplified comparisons, providing a quick overview of diverse styles. With the flexibility to adjust prompts, this tutorial empowers artistic self-expression, transforming face portraits into personalized visual statements.