Sprite Animation Generation - Background Removal 2

Previously, we looked at bunch of background removal options to use after we generate a sprite animation using video gen AI tools. There were some options that were promising but still fell short in edge cases. Well, background removal is all about the “edges,“ so I’m still in search of better options.

TL;DR: Using Flux Kontext Pro smartly to remove background wins.

Input Image

---

Anime-Seg

One promising option I found was skytnt/anime-seg which was trained on drawn images. Compared to matting models that were trained on real photos, this does substantially better.

However, it still falls short in edge cases and because it was trained on line-art style images, it doesn’t doesn’t provide ultra crisp zones.

The source code and model is all available, so we can probably fine-tune it to better work for the pixel arts domain.

---

Flux Kontext Pro Hack

Flux Kontext Pro focuses on image understanding and the main selling point is its ability to Edit photos surgically.

So you can ask the model to do things like:

Prompt: Change the background to black

This produces almost flawless switching out of the background.

So, now we have two version.

• \(A\): Original with white background.

• \(B\): Black background version generated by Flux Kontext Pro.

If we perform \(A - B\), we get a nice segmentation mask that looks like this.

Now, we invert this image, and viola, we have a working segmentation mask.

Once you apply this alpha mask on B, you will get transparent version without any seams.

Here is a view of this sheet with some colored background for demonstration.

I haven’t tried this on animated sprites, but it seems very promising so far.

---

Why not train our own model? Perhaps, we should train our own model though Flux Kontext seems proficient enough. I would love to fork anime-seg repo and train to segment pixel arts.

—Sprited Dev