SD & A1111 Tutorial 5: Latent Couple

1 of 10 next end

set default image size: small | medium | wide

1 of 10 next end

show original thumbnails

by Logically

Gallery

Stable Diffusion and Automatic1111 Tutorial Series (journal link)

Covered in this tutorial

• Latent Couple: what is it, how to install it, how to use it
• Example render

What is Latent Couple

The last tutorial described the problem of generating renders with multiple subjects and the cut-and-paste method as one solution. This tutorial covers a different solution called Latent Couple.

Latent Couple (or stable-diffusion-webui-two-shot) is an A1111 extension that allows you to segment an image canvas into regions and apply a separate prompt for each region. Notably, this allows you to define a render with multiple characters by giving each character a defined region of the image and a separate dedicated prompt.

Pros:

• It is possible to create a multi-subject scene with a single render, without needing to copy-and-paste, and no need to edit renders or use img2img.
• If combined with something like OpenPose, it gives you a huge amount of control over the scene and subjects.
• This method produces renders where subjects can touch and interact with each other in ways that are hard to achieve with cut-and-paste. To get this point across, the example render is a couple hugging. This is technically possible to achieve with cut-and-paste but it would be more difficult and likely won’t look quite as natural.

Cons:

• The plugin setup and prompt structure can be tricky and finicky to use. You can get into situations where it doesn’t work at all and its not really clear why.
• Each additional region in the render (including the white background region) increases the overall render time.
• The more regions you have, the less you will be able to upscale. The resolution at which upscaling slows to a crawl or causes out of memory errors is lower.
• The built-in methods of creating masks are not very good in my opinion. I believe it is better to create your masks separately from scratch. This means using some imaging editing software. It doesn’t have to be GIMP, but that is what I am using.

Installing Latent Couple

https://github.com/ashen-sensored/stable-diffusion-webu...
This is the GitHub link to the Latent Couple extension, but you don’t need to install it from here. I am linking this because it has some good documentation.

You can install LatentCouple directly within A1111. Go to the “Extensions” tab, the “Available” tab under that, search “Latent Couple” and Install from there. Image 2A shows this in A1111. A reload should add a Latent Couple section to the Generation tab.

Note on Composable Lora

https://github.com/a2569875/stable-diffusion-webui-comp...
I’ve also sometimes used another tool called Composable Lora. It can be used along with Latent Couple for the purpose of using different character LoRAs. However, I hesitate to recommend it since I have been able to make mixing LoRas without it and I’m not sure how necessary it is. I will say Composable Lora does make the render differnet, but maybe not necessarily in a good way. I won’t cover it for the rest of this tutorial, but I will at least mention it here as an option if you are having trouble with Latent Couple alone. You can install it in A1111 the same way.

https://www.kombitz.com/2023/04/16/how-to-use-latent-co...
This tutorial on Latent Couple and Composable Lora is what I originally started with when I first tried these out. Parts of it were really helpful but I had to figure out a lot of details on my own.

Using Latent Couple: making a mask

Latent Couple requires using a mask, which is basically an image with different colors used to define the separate regions. The plugin has a couple tools to make a mask within A1111. You can use these if you can make them work for you, but I would recommend making your own in a separate editing program.

I will at least go over the in-plugin ways to make masks:
• The “Rectangular” tab gives a simple way to define masks, but I could never really get it to work for me. The tutorial I linked above goes into detail on this approach. Maybe it will work for you.
• The “Mask” tab has an option “Create blank canvas” which supposedly lets you draw a mask in the UI. This feature is super laggy and slows down my browser to the point where it crashes or I have to kill the process. Strongly not recommended.

The video on this post is my editing in GIMP to create the mask. In this case, I used a separate render 1A as a reference to trace out my mask. I will sometimes do this approach of creating a separate reference render just for the purpose of creating the mask by hand. This is totally optional and depending on your skill level, unnecessary. In this situation, I was recording a video that needed to be short, so I needed a reference to make it fast.

For editing, you can use whatever you want, but I like to use GIMP so that I can define separate layers for each color. This makes it a lot easier to go back and make changes to the mask, which I do a lot. Similarly to writing a prompt, the mask goes through a LOT of trial and error. I’ll often go maybe 10-20 iterations on a mask before it is perfect, but that is partially because I tend to use a high number of subjects.

The colors you use for the mask mostly don’t really matter, except that they do influence the ordering of the regions, based on the alphabetical order of the RGB hex values. For example, blue (#0000ff) is before green (#00ff00) is before red (#ff0000). If you are creating an image with multiple subjects that are positioned in a clear left-to-right order, it can be helpful to have the prompt order be the same as the subject order.

I typically pick colors like this when making a mask based on how many subjects:
• 2 subjects – blue, red
• 4 subjects – blue, green, red, yellow
• 6 subjects – blue, green, cyan, red, magenta, yellow
• 8 subjects – blue, green, cyan, purple, red, magenta, orange, yellow

You can see this mask color ordering in these examples (warning NSFW):
Group Shower Buddies and Thank You for 1000 Watchers!!!

A region can be in different parts. As long as they are the same color, they are treated as the same region. Parts of the mask that are white are used for the background and are treated as its own region.

Using Latent Couple: using the mask

In Latent Couple
• Check “Enabled”
• Upload your mask to the “Mask” tab. You do have the option to edit the mask here, but I would do any edits on the original program you used.
• Click “I’ve finished my sketch”. You should see it calculate the different regions.
• Click “Prompt Info Update” (Warning: this overwrites what you have in the positive prompt)
• The positive prompt should now contain an “AND” separator for each region.

You have the option of whether to work on your prompt in the regular prompt window, or in the Latent Couple UI. I personally find it much easier to use the regular prompt window for prompt tuning.

There’s a minor usability issue when it comes to reuploading a mask. Sometimes when working on a prompt I decide that part of the mask needs to change. When I edit a mask and reupload it, in order for that new mask to apply to the render, you HAVE to click “I’ve finished my sketch” and “Prompt Info Update” again; WHICH OVERWRITES THE POSITIVE PROMPT.

Whenever I have to edit and reupload the mask, I usually just Ctrl+C copy the entire contents of the positive prompt, and Ctrl+V paste it in after clicking “Prompt Info Update”. Because of this, you may decide to do all your prompt editing in the Latent Couple view. But I really dislike it because the text boxes are much smaller and you have to click twice instead of once every time you iterate on the render.

If you do happen to lose the prompt, you can get it back from a previous render. “PNG Info” tab lets you upload a previous render and “Send to txt2img”. Doing this does not overwrite the Latent Couple mask settings.

I’ve not experimented with the “Alpha Blend” and “Area [X] Weight” settings. I always keep those at the default 0.2 and 1.0 values. You can experiment with these to get different results, but the defaults work well for me.

Example Render

The sample render is 2 foxes hugging. They are of different fur color and gender, which is difficult to achieve with a regular prompt.

Image 1A is the reference image used to trace the mask. Image 1B is the Latent Couple mask (for reproducing this render, you’ll need to download this one). Image 1C is the final render created using the mask.

The full prompt and settings are in the text file, but I’m showing just the positive prompt here for demonstration purposes. The positive prompt is split by “AND” into the 3 sections to cover the 3 regions: first the background, then the orange fox, then the black fox. If the blue/red colors on the mask were swapped, I would also have to swap them in the prompt.

"

(public park:1.2), trees, autumn, buildings, (bench:0.7), masterpiece, high quality, realistic, detailed background
AND
male, (orange fox:1.1), anthro, furry, (hugging:1.2), smiling, (mouth open:1.1), (holding each other:1.1), (pink shirt:1.1), tail, hand on shoulder,
masterpiece, high quality, realistic,
AND
female, (black fox:1.1), anthro, furry, (hugging:1.2), smiling, (mouth open:1.1), (holding each other:1.1), (blue shirt:1.1), tail, hand on shoulder,

Images 2B, 2C, 2D show the A1111 prompt setup used to create the final example render.

No additional edits were made on this image after it was rendered.

Keywords

male 1,191,719, female 1,081,877, fox 247,835, ai generated 22,678, red fox 9,105, hugging 5,614, stable diffusion 4,974, park 4,329, tutorial 617, black fox 314, tutorials 88

Details

Type: Picture/Pinup

Published: 6 months, 2 weeks ago

Rating: General

MD5 Hash for Page 1... Show Find Identical Posts [?]