Recently ControlNet got a lot of popularity in the open-source community. I am interested in running it on my Windows 11 workstation via the StableDiffusion. This article assumes we have already setup the StableDiffusion libary on the workstation. If you haven’t yet, don’t worry, you can find a step-by-step guide in my other article.
The first thing we need to do is launch the StableDiffusion Web Interface and install the ControlNet extension.
Then, we need download the ControlNet weights. To do this we will go to HuggingFace’s ControlNet repo here. And we will download all different model weights for each type of inputs. Downloads the model .pth files and put into the ControlNet extention/model directory.
Now you need to refresh the model, enable preprocessing to “OpenPose”, and select the controlnet_sd15_openpose.pth in the model. Then upload an image to use as the seed pose reference. Click “Generate”, there you go, Simu’s pose is not transferred, while the image follows my prompt — “new york street, yellow cab, male fashion model, drinking coffee”!
I also tried seed the pose using a synthetic CG image, and changed the batch size to 8. It works pretty well.
