Tag Archives: stable diffusion webui

Build auto end-to-end image-to-image transformer using stable diffusion web framework

Posted on by Leave a comment

There are many articles, posts or YouTube video teaching how to use stable diffusion web UI framework, https://github.com/AUTOMATIC1111/stable-diffusion-webui/tree/master, to replace face, background, cloth, style, etc. The interactive UI is easy to explore the capability of stable diffusion, e.g. how different diffusion model effects on the style of generated image, how to use inpainting to replacing face, cloth, background. But we are interesting to build auto end-to-end pipeline to do model transform, i.e. keeping cloth while replacing model and their pose. Here is the main recipe:

  • Choose a base stable diffusion model based on the style your preferred. There are a lot of open-source models you can find in https://civitai.com/
  • Segment and mask cloth. You can use Meta segment anything, https://segment-anything.com/
  • Select a pose template. You can choose image that can extract pose. OpenPose can do pose extraction, https://github.com/CMU-Perceptual-Computing-Lab/openpose.
  • Align the cloth to pose template. The step is very important. If the alignment is good, perfect image is generated. Otherwise, the generated image is weird something.

Connect all the modules above together, end-to-end pipeline can be built. Input a cloth image & pose template image, then output try-on cloth image with human model. Of course, the method works best on mannequin image transforming to human model. But for general challenging try-on, it cannot.

Correctly install sd-webui-train-tools extension in stable diffusion webui to train Lora

Posted on by Leave a comment

The easy practice to train Lora model & evaluate it is to using stable diffusion webui, https://github.com/AUTOMATIC1111/stable-diffusion-webui/tree/master, and install extension sd-webui-train-tools, https://github.com/liasece/sd-webui-train-tools. But after installed extension, following instruction to train lora model, there is a error happened can cause lora training failed. The error is when importlib networks.lora, it reports networks.lora not a module.

I spend some time to investigate the reason and try fix it. Google cannot find a solution. From the error, it looks that module path is not in system path. But after append module path into system path. The error still is there. After debug, finally find it is a bug in https://github.com/liasece/sd-webui-train-tools/blob/main/liasece_sd_webui_train_tools/train.py. In the following function, sub_module=”library” is wrong. Correct value is None. After set sub_module=None (or remove the argument because default is None), Lora training starts

def train(cfg: ArgStore) -> None:
    args = cfg.create_args()
    with pc.PythonContextWarper(
            to_module_path= os.path.abspath(os.path.join(os.path.dirname(__file__), "sd_scripts")), 
            path_include= os.path.abspath(os.path.join(os.path.dirname(__file__), "..")), 
            sub_module="library",
        ):
        # begin training
        train_network.train(args)

Correct version:
def train(cfg: ArgStore) -> None:
    args = cfg.create_args()
    with pc.PythonContextWarper(
            to_module_path= os.path.abspath(os.path.join(os.path.dirname(__file__), "sd_scripts")), 
            path_include= os.path.abspath(os.path.join(os.path.dirname(__file__), "..")), 
            sub_module=None,
        ):
        # begin training
        train_network.train(args)