To summarize this week’s progress: we’re knee-deep in building SpriteDX, a Unity editor plugin that generates fully animated characters from simple text prompts. Next up: crafting demo sprites that really show off what this baby can do.

Previous Posts

We faced two problems—Pixel Grid Misalignment and Background Removal Imperfection.

The Pixel Grid Misalignment issue is a bit of a subjective snag. Our “off-grid“ pixels still look like pixel art—but they don’t snap to the grid perfectly, which can trip up any manual touch-ups.

On the flip side, those tiny sub-pixel offsets let us draw eye glints and fine details that feel alive. If we force everything onto the native grid, you either get glaringly bright pixels or lose those sparks altogether.

Now, pixel art purist would be angered to see a sprite generation tool that doesn’t generate “true“ pixel arts, so yeah, we need to think through this further and consider training a model that will digitize this into native pixel resolutions as discussed here.

IMO, Background Removal Imperfection is a more immediate issue. State of the art video generation models like Seedance 1 Pro does not support transparency and we need an automated way to make the background transparent.

Inspyrenet

We tried Inspyrenet custom node in Comfy, but they were still not able to capture enclosed areas properly.

Segment Anything

Segment Anything was also not able to capture this triangle region.

Segment Anything 2 Video

Version 2 takes in the whole video and is quite reliable but still fails on the triangle region below the chin.

Robust Video Matting

RVM looked promising for webcam dataset but did not perform very well on the sprite dataset we had.

BRIAAI Matting

BRIAAI Matting was much better on most of the frames, but didn’t work so well in some others.

Color Keying + SAM2

One of the potential issue is that we are using white background on a character wearing white costume. I’m trying a color key approach.

So, I regenerated the video after coloring the background purple #FF00FF.

BRIAAI node still failed right below the arm pit.

But, SAM2 was able to produce correct segmentation even in enclosed zones.

Licensing wise, SAM2 is Apache-2.0 and is available for commercial use.

However, one downside is that it seems to create segmentations with rounded corners which are detrimental to pixel arts.

Either we will want to post process this mask or find alternative models.

Conclusion

I couldn’t find one perfect model for this. Color-key + SAM2 video was the best option so far but it is not perfect. For the time being, we will choose best model from here and use it for the SpriteDX prototype.

—Sprited Dev 🌱

Sprite Animation Generation - Background Removal