SD & A1111 Tutorial 7: Segmentation Mask

1 of 11 next end

set default image size: small | medium | wide

1 of 11 next end

show original thumbnails

by Logically

Gallery

Stable Diffusion and Automatic1111 Tutorial Series (journal link)

An Update

This will be my last tutorial in the series for now. I’ve now covered most of the big topics I wanted in this series, and I am happy with how they turned out.

Covered in this tutorial

• What are segmentation masks
• Segmentation in ControlNet
• Example render

ControlNet refresher

https://inkbunny.net/s/3397479 My previous tutorial goes into detail on what is ControlNet and how to set it up. There are many included control types, of which I previously covered OpenPose.

Segmentation

Segmentation is another ControlNet control type. Many of my previous tutorials focus on how to manipulate the subjects in your renders. This one is more focused on manipulating the non-subject elements, including the background and inanimate objects.

https://en.wikipedia.org/wiki/Image_segmentation (optional reference)

In computer vision, segmentation is a process of dividing an image into regions and assigning categories to those regions. The categories are typically types of objects that will commonly appear in a photograph. Specific colors are used to represent each category in a segmentation mask. These masks can then be passed into other programs for various purposes.

ControlNet can use a segmentation mask to manipulate your render by attempting to place objects in the scene based on the mask.

Pros:

• This is the only method I know that gives you control over the background and objects. For exterior scenes, this is very useful for manipulating the scenery.
• Similarly, this can also be used to place inanimate objects in the scene. I have found this useful for things like showerheads and toilets, which the subjects are not directly interacting with.
• Segmentation can give you influence over the number of subjects in a scene and their placement.

Cons:

• Making segmentation masks can be tricky and can require a lot of trial and error.
• Segmentation does not give you a lot of control over the pose of a subject like OpenPose can.

Segmentation mask preprocessor

There are 2 ways to make a segmentation mask: using a program or making it by hand.

Typically, an AI program is used to process an image to identify regions and produce a segmentation mask. ControlNet does have the ability to preprocess an image into a segmentation mask. The red explosion icon in ControlNet runs the preprocessor.

https://groups.csail.mit.edu/vision/datasets/ADE20K/ (optional reference)

I usually use seg_ufade20k or seg_ofade20k as the preprocessor, which stands for Oneformer and Uniformer ADE20K. ADE20K is a dataset of images with segmentation masks that can be used for training AI programs. You don’t really need to know this to use ControlNet, it’s just useful to have context.

If I use the preprocessor, I’m typically editing it heavily. In some cases it’s better to just create it by hand from scratch.

Creating a mask by hand

https://docs.google.com/spreadsheets/d/1se8YEtb2detS7Ou...

This link is the most important one. Save or bookmark this. It contains a set of color segmentation categories that are recognizable by ControlNet. When you create or modify a mask, you will need to use these colors to indicate what kind of object you want to go where.

When I make a mask by hand, I’m creating it in GIMP. I usually make separate layers for each color category.

Tip: While making this tutorial, I discovered that the “person” category (#96053D) that I normally use for the subjects was not working as well as it usually does. Maybe it was an issue with my prompt, but it kept putting a weird tree stump in place of the fox. In desperation, I tried the “animal” category (#FF007A) which for this render happened to work out perfectly. If you are rendering furry characters, I would just keep this in mind. The “person” should normally work but sometimes it might not.

Using a segmentation mask in ControlNet

If you are creating your own mask, you need to enable “Mask Upload” in the UI, which creates a separate place to upload your mask.

I would also suggest using “My prompt is more important” for Control Mode. Unless your mask is perfectly defined, anything stronger will have a tendency to overinfluence your render and make it not good.

There are some other settings but I don’t normally mess with them that much.

My renders using this method

https://inkbunny.net/submissionsviewall.php?rid=5b40600...

(Warning: NSFW) This search should include every post I’ve made with a segmentation mask. I always include the mask for reference. Notable examples:

• Group Shower Buddies and Urinal Wolves #2 are good examples that utilizes the mask to place showerheads and urinal in the right places
• Lion and Bunny: Cuckold is a good interior scene that includes windows, bed, and a chair
• Bottomless Bear Bunch and Bear and Tiger: Riverside Rut are good exterior scenes that compose good landscapes

Example render

I wanted to demonstrate a video of making a mask from scratch, so the scene had to be fairly simple. An exterior shot in the wilderness is a great use case for segmentation, since many of the color categories are well suited to it.

The categories I used were: grass, river, trees, mountain, sky, house, animal. Just for the purpose of making the video, I created a palette of these color codes that I could reference. I’ve included this palette as an image as 3B.

The main thing I want to call attention to is the inclusion of a cabin. Normally, I find it hard to put a single building in the background of a scene like this. I find that it tends to want to make the scene to be from inside a cabin. With segmentation, I can force the cabin to be in the background instead of the foreground.

The video shows me making the segmentation mask 2A. Just for demonstration purposes, I horizontally flipped the mask to make 2B. I used both versions of these masks with the same prompt to make renders 1A and 1B. Just for fun, I also created another render 1C without using the segmentation mask. You can see that this one actually looks mostly fine but is missing the cabin.

Image 3A shows an overlay of the final renders on top of the segmentation masks to show how much the mask influences the render. There are a couple ways that the render deviated, most notably the bottom corner is grassy in the render but water in the mask. There are a few other places where the regions mostly line up but not perfectly. I’m actually fine with this, since the mask was put together quickly.

You should be able to use the same mask to reproduce the render. Images 4A and 4B show my settings in A1111. Prompt and settings for render:

"

Positive prompt:
solo, (orange fox:1.1), anthro, furry, (colorful shirt:1.1), shorts, black hands, (sitting on grass:1.1), happy,
(river:1.1), trees, mountains, sky, flowers, cabin, masterpiece, high quality, realistic, detailed background

Negative prompt:
bad quality, deformity,

Steps: 30, Sampler: Euler a, Schedule type: Automatic, CFG scale: 7, Seed: 5, Size: 800x600, Model hash: b965aee5a3, Model: indigoFurryMix_v120Hybrid, Denoising strength: 0.4, ControlNet 0: "Module: seg_ofcoco, Model: controlnetPreTrained_segV10 [b9c1cc12], Weight: 1, Resize Mode: Crop and Resize, Low Vram: False, Processor Res: 512, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: My prompt is more important, Hr Option: Both, Save Detected Map: True", Hires upscale: 2.56, Hires upscaler: R-ESRGAN 4x+, Downcast alphas_cumprod: True, Version: v1.9.4

Keywords