InstantID: Zero-shot Identity-Preserving Generation in Seconds

Qixun Wang¹² · Xu Bai¹² · Haofan Wang^12* · Zekui Qin¹² · Anthony Chen¹²³ · Huaxia Li² · Xu Tang²

¹InstantX Team · ²Xiaohongshu Inc · ³Peking University

^*corresponding authors

InstantID is a new state-of-the-art tuning-free method to achieve ID-Preserving generation with only single image, supporting various downstream tasks.

Release

• [2024/02/01] 🔥 We have supported LCM acceleration and Multi-ControlNets on our Huggingface Spaces Demo! Our depth estimator is supported by Depth-Anything.
• [2024/01/31] 🔥 OneDiff now supports accelerated inference for InstantID, check this for details!
• [2024/01/23] 🔥 Our pipeline has been merged into diffusers!
• [2024/01/22] 🔥 We release the pre-trained checkpoints, inference code and gradio demo!
• [2024/01/15] 🔥 We release the technical report.
• [2023/12/11] 🔥 We launch the project page.

Demos

Stylized Synthesis

Comparison with Previous Works

Comparison with existing tuning-free state-of-the-art techniques. InstantID achieves better fidelity and retain good text editability (faces and styles blend better).

Comparison with pre-trained character LoRAs. We don't need multiple images and still can achieve competitive results as LoRAs without any training.

Comparison with InsightFace Swapper (also known as ROOP or Refactor). However, in non-realistic style, our work is more flexible on the integration of face and background.

Download

You can directly download the model from Huggingface. You also can download the model in python script:

from huggingface_hub import hf_hub_download
hf_hub_download(repo_id="InstantX/InstantID", filename="ControlNetModel/config.json", local_dir="./checkpoints")
hf_hub_download(repo_id="InstantX/InstantID", filename="ControlNetModel/diffusion_pytorch_model.safetensors", local_dir="./checkpoints")
hf_hub_download(repo_id="InstantX/InstantID", filename="ip-adapter.bin", local_dir="./checkpoints")

Or run the following command to download all models:

pip install -r gradio_demo/requirements.txt
python gradio_demo/download_models.py

If you cannot access to Huggingface, you can use hf-mirror to download models.

export HF_ENDPOINT=https://hf-mirror.com
huggingface-cli download --resume-download InstantX/InstantID --local-dir checkpoints

For face encoder, you need to manually download via this URL to models/antelopev2 as the default link is invalid. Once you have prepared all models, the folder tree should be like:

  .
  ├── models
  ├── checkpoints
  ├── ip_adapter
  ├── pipeline_stable_diffusion_xl_instantid.py
  └── README.md

Usage

# !pip install opencv-python transformers accelerate insightface
import diffusers
from diffusers.utils import load_image
from diffusers.models import ControlNetModel

import cv2
import torch
import numpy as np
from PIL import Image

from insightface.app import FaceAnalysis
from pipeline_stable_diffusion_xl_instantid import StableDiffusionXLInstantIDPipeline, draw_kps

# prepare 'antelopev2' under ./models
app = FaceAnalysis(name='antelopev2', root='./', providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
app.prepare(ctx_id=0, det_size=(640, 640))

# prepare models under ./checkpoints
face_adapter = f'./checkpoints/ip-adapter.bin'
controlnet_path = f'./checkpoints/ControlNetModel'

# load IdentityNet
controlnet = ControlNetModel.from_pretrained(controlnet_path, torch_dtype=torch.float16)

base_model = 'wangqixun/YamerMIX_v8'  # from https://civitai.com/models/84040?modelVersionId=196039
pipe = StableDiffusionXLInstantIDPipeline.from_pretrained(
    base_model,
    controlnet=controlnet,
    torch_dtype=torch.float16
)
pipe.cuda()

# load adapter
pipe.load_ip_adapter_instantid(face_adapter)

Then, you can customized your own face images

# load an image
face_image = load_image("./examples/yann-lecun_resize.jpg")

# prepare face emb
face_info = app.get(cv2.cvtColor(np.array(face_image), cv2.COLOR_RGB2BGR))
face_info = sorted(face_info, key=lambda x:(x['bbox'][2]-x['bbox'][0])*x['bbox'][3]-x['bbox'][1])[-1]  # only use the maximum face
face_emb = face_info['embedding']
face_kps = draw_kps(face_image, face_info['kps'])

# prompt
prompt = "film noir style, ink sketch|vector, male man, highly detailed, sharp focus, ultra sharpness, monochrome, high contrast, dramatic shadows, 1940s style, mysterious, cinematic"
negative_prompt = "ugly, deformed, noisy, blurry, low contrast, realism, photorealistic, vibrant, colorful"

# generate image
image = pipe(
    prompt,
    image_embeds=face_emb,
    image=face_kps,
    controlnet_conditioning_scale=0.8,
    ip_adapter_scale=0.8,
).images[0]

To save VRAM, you can enable CPU offloading

pipe.enable_model_cpu_offload()

Speed Up with LCM-LoRA

Our work is compatible with LCM-LoRA. First, download the model.

from huggingface_hub import hf_hub_download
hf_hub_download(repo_id="latent-consistency/lcm-lora-sdxl", filename="pytorch_lora_weights.safetensors", local_dir="./checkpoints")

To use it, you just need to load it and infer with a small num_inference_steps. Note that it is recommendated to set guidance_scale between [0, 1].

from diffusers import LCMScheduler

lcm_lora_path = "./checkpoints/pytorch_lora_weights.safetensors"

pipe.load_lora_weights(lcm_lora_path)
pipe.fuse_lora()
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

num_inference_steps = 10
guidance_scale = 0

Start a local gradio demo

Run the following command:

python gradio_demo/app.py

or MultiControlNet version:

gradio_demo/app-multicontrolnet.py 

Usage Tips

• For higher similarity, increase the weight of controlnet_conditioning_scale (IdentityNet) and ip_adapter_scale (Adapter).
• For over-saturation, decrease the ip_adapter_scale. If not work, decrease controlnet_conditioning_scale.
• For higher text control ability, decrease ip_adapter_scale.
• For specific styles, choose corresponding base model makes differences.
• We have not supported multi-person yet, only use the largest face as reference facial landmarks.
• We provide a style template for reference.

Community Resources

Replicate Demo

• zsxkib/instant-id

WebUI

• Mikubill/sd-webui-controlnet

ComfyUI

• ZHO-ZHO-ZHO/ComfyUI-InstantID
• huxiuhan/ComfyUI-InstantID

Windows

• sdbds/InstantID-for-windows

Acknowledgements

• InstantID is developed by InstantX Team at Xiaohongshu Inc, all copyright reserved.
• Our work is highly inspired by IP-Adapter and ControlNet. Thanks for their great works!
• Thanks Yamer for developing YamerMIX, we use it as base model in our demo.
• Thanks ZHO-ZHO-ZHO, huxiuhan, sdbds, zsxkib for their generous contributions.
• Thanks to the HuggingFace gradio team for their free GPU support!
• Thanks to the ModelScope team for their free GPU support!
• Thanks to the OpenXLab team for their free GPU support!
• Thanks to SiliconFlow for their OneDiff integration of InstantID!

Disclaimer

The code of InstantID is released under Apache License for both academic and commercial usage. However, both manual-downloading and auto-downloading face models from insightface are for non-commercial research purposes only accoreding to their license. Users are granted the freedom to create images using this tool, but they are obligated to comply with local laws and utilize it responsibly. The developers will not assume any responsibility for potential misuse by users.

Star History

Cite

If you find InstantID useful for your research and applications, please cite us using this BibTeX:

@article{wang2024instantid,
  title={InstantID: Zero-shot Identity-Preserving Generation in Seconds},
  author={Wang, Qixun and Bai, Xu and Wang, Haofan and Qin, Zekui and Chen, Anthony},
  journal={arXiv preprint arXiv:2401.07519},
  year={2024}
}

For any question, please feel free to contact us via [email protected] or [email protected].