The author of ControlNet's new work is a hit: P photos can be changed into backgrounds without asking for help, and AI lighting is perfectly integrated

The new work by the author of ControlNet is so fun to play that it received 1.2k stars just after it was open sourced.

IC-Light is used to manipulate image lighting effects, the full name is lmposing Consistent Light.

The gameplay is very simple:

Uploading means that the system will automatically separate the characters and other subjects, select the light source position, fill in the prompt words, and you can integrate into the new environment without any flaws!

Hurry up and do Wong Kar-wai style lighting:

Don’t like it?

It doesn't matter, it only takes a few minutes to switch to the natural light coming in from the window.

Currently, IC-Light provides two types of models: text conditional relighting models and background conditional models.

Both models require a foreground image as input.

Given that Controlnet was so fun before, IC-Light attracted a lot of attention when it appeared this time, and some netizens quickly made the ComfyUI plug-in.

(Doubtful, everyone works so hard without sleeping??)

Whether it is the expectations or the experience after use, netizens are very generous Gao:

Nice! Can’t wait to get started and play hehehehe.

Who can help me change the background of this picture?

From ancient MCN to Tieba and now Xiaohongshu, in every era, there is no shortage of help posts like "Can anyone help me change the background?"

But the help from enthusiastic netizens often looks like this:

is outrageous.

But to be honest, this kind of demand does not only exist among ordinary people like you and me. E-commerce companies often have similar needs when making product posters.

With IC-Light, everything seems to have become easier.

Upload the original main image + select the light source position + prompt word, and you’re done.

Let’s see the effect:

Such an original picture of the Buddha statue, add the prompt words "Buddha statue, detailed face, sci-fi RGB glow, cyberpunk", and then select "Light from left" Hit from the side."

You will get a brand new finished product:

It is suitable even for daily scenes.

The final effect looks more natural to the naked eye:

According to the evaluation shared by netizens, it is also suitable for animation scenes...

Behind the Technology

As mentioned before, IC-Light now provides two types of models, both of which require foreground images as input.

One type is the text conditional relighting model.

To put it simply, users can complete the generation by entering prompt words.

For example, if you input "left light", "moonlight", etc., the model will use these prompt words and initial latent variables to generate images that meet the requirements and characteristics.

The other type is the background condition model.

This one is simpler and does not require complex prompt words. The model combines the background prompt information to perform different styles of lighting changes on the foreground objects.

The technical principle behind it is to ensure that the model output is consistent under different light source combinations through the consistency of the latent space, so that various lighting effects can be stably synthesized.

details as follows:

In the HDR space, the light transmission of all lighting is independent of each other. The appearance mixing effect of different light sources is mathematically (that is, in an ideal state) consistent with the appearance under the direct action of multiple light sources. .

Take the lighting stage of the above picture as an example. The two images from "Appearance Mixing" and "Light Source Mixing" are consistent, (ideal situation below, mathematically equivalent in HDR space).

Therefore, when training the relighting model, the researchers used a multilayer perceptron (MLP) in the latent space to make the combination and transmission of different light sources consistent and used to guide the generation Effect.

The result is a highly consistent relighting effect.

Because the model uses latent diffusion technology, learning and relighting operations can be implemented within the latent space, resulting in highly consistent effects under various lighting conditions.

These results are very consistent - even though the model does not use the normal map data directly when training, the different relightings can be merged into normal maps.

Look at the picture below, from left to right are the input, model output, relighting, split shadow image and merged normal map.

Interested friends can go to the address below to try it out~

GitHub direct train: https://github.com/ lllyasviel/IC-Light?tab=readme-ov-file.