Sprite DX - Stage 4 - Automating Background Removal Process

Yesterday, we automated process of splitting shots and extracting clean loops. Next is background removal.

Ingredients

We start with those loop videos that we created in previous post.

Idle Loop

Run Loop

---

Approach

We explored few approaches before (1), (2), and landed on an approach that uses FLUX.1 Kontext to replace the background color to black and subtracting the original to get a alpha mask of the background. We have not extensively tested it but initial results showed promise.

So, in this section, we create a ComfyUI custom node that will repeat this process for every frame of the input video.

• We first load the video using Load Video node from VHS, and resize the all frames into 256×256.

• Then, we create a sprite sheet of 4 rows and 4 columns (total 1024×1024 pixels, which is one of the native resolutions for FLUX.1 Kontext).

• We then pass the image to FLUX.1 Kontext with a prompt “Chagne the background to black.“

• Then we subtract original whiteBG sheet and blackBG sheet to produce a mask.

• We post process the mask by applying a level.

• Then, apply the alpha mask

• Break the grid into the frames and slice empty frames.

• Then, we save it as a webp.

Result

Gotta go.

—Sprited Dev