Sprite Animation Generation - Pixel Alignment Problem

Background

We tried generating animation of pixel art character in previous post and faced some issues when turning generated video into pixel arts of native pixel dimensions.

Here is the generated video from Seedance 1 Pro.

Let’s take a frame and study it further.

Problem

If you blow up this frame and zoom in, you can see that the these imaginary pixels don’t actually align. In a way, they only really give an illusion of pixel art rather than providing perfect pixel alignments.

So, what ends up happening is when we downsample it to native dimensions using nearest neighbor sampling, we don’t get that crisp clear edges. We get something like:

The outline’s are blurrier and high frequency signals like eye lights are gone. Overall the character feels like it was antialiased.

---

Food For Thought: Retro Pixel Art v.s. Non-Regularized Pixel Art

Before we navigate further, we must ask whether perfect pixel alignment is really necessary or even desired. Retro Pixel Art is, in a way, an art form that’s been developed to cope with limited amount of video rams in early game consoles. That’s not really a concern in modern day. Perhaps, this new style of non-regularized pixel art form has its own potential.

---

Possible Solution

Digitization of non-regular pixel art is not an easy task. There may be some solutions already available, so probably worth looking around more, but I couldn’t find good ones that are truly able to tackle this at satisfying quality.

One idea is to train or fine-tune a model to do this. We collect diverse set of beautiful pixel arts \(P\).

Then, we will transform them via various means to create a variant of an image that mimics “imperfect“ pixel art \(P'\). Generating this imperfect pixel arts will most likely be a forward pass of image-to-image pipeline that uses \(P\). We can also include some upscaled images from PixelPerfectV4 since that is known to give out high resolution images based on pixel art images.

Given that dataset, we train the model to go from \(P'\) to \(P\). That way the model can go from imperfect to pixel perfect.

---

Anyway, that’s it for today. I’ll investigate on this further in following days.

—Sprited Dev