SpritedDX’s vision is to make a tool that will create animated and working platformer characters from text prompts.

In yesterday’s post, we created Comfy workflow for Stage 1 to generate character reference.

Today, we strive to build Stage 2. Stage 2 will animate the character using video gen AIs.

What I will do

I will create a Comfy workflow that will generate animated frames of run cycle.

Prerequisite

Fal.ai: You need Fal.ai account and need some credits added to your account.
- You can use other providers like Scenario, but setup is going to be more manual.
Comfy UI: I’m using ComfyUI to create the workflow.

Reference image character image generated in Stage 1.

First, we use the ComfyUI Manager’s Custom Node Manager to install ComfyUI-fal-API.

You have to follow the instructions to configure it with your Fal AI API keys.

We can then add Seedance Image-to-video (fal) node and type in some prompt.

Flow Summary

Load Image node to load the reference image
Seedance Image-to-Video node to generate the video from the reference image and a prompt. This outputs a video URL.
Now, use use Load Video from URL (fal) node to load the videos into frames.
Finally, we use Video Combine (VHS) node to construct a gif or mp4.

Still need to do some more work to tune the prompts and make the character move in place. Logging off now. I will continue later.

Update: Had to read through the Seedance 1.0 Paper to understand the prompt format. I have a pretty good solution here: https://blog.sprited.app/designing-multi-shot-prompts-for-sprite-animation-generation

—Sprited Dev 🌱