In this blog post, we explore ways to split 5 second video into multiple shots.

Previously…

In previous post, we designed prompt format for Seedance 1.0 Pro use in generating animation frames. We are using the following multi-shot prompt.

[Scene Description] Character is a game sprite character for game called “Machi.” Her name is “Eliana.” The video has no camera movements.

[SHOT 1] Intro: Camera static full body shot and pixel art character says “hi.” [CUT]

[SHOT 2] Idle Loop: Camera static full body shot and the pixel art character is facing slightly right (+x) looking straight right (+x) and shows sprite animation loop for “idle” state, breathing in and breathing out, on a pure white background. [CUT]

[SHOT 3] Run Cycle (in-place, +x): Camera LOCKED on the pixel art character running in positive x direction, and character starts showing sprite animation loop for “run” state on a pure white background. [CUT]

[Tags] #character-animation #角色动画

#game-sprite #游戏精灵

#platformer #平台跳跃

#machi #Machi

#cute #可爱

#pixel-art #像素艺术

#1girl #少女

#subject-only #主体突出

#character-only #纯人物

#side-scroller #横版卷轴

#platformer #平台游戏

#idle-loop #待机循环

#run-cycle #跑步循环

#pure-white-background #纯白背景

It describes overall scene and then goes into 3 different shots—saying hi, idle loop, run loop. As a result, we get generations like the one you see here.

Problem

The problem is that this generated video clip now contains multiple shots (aka “cuts“). And to use it as a character animation sequence, we need to break the video down to multiple videos each containing one shot.

Let’s explore some options for splitting these shots into separate videos. From what I read from Seedance paper, they created a model that will split the shots. So, there may be open-source models that already do this.

PySceneDetect

PySceneDetect seems to be one of the most popular solutions here.

I vive coded a script to feed in the mp4 file and split into multiple shots.

We are using threshold value of 8.0.

poetry run python ./scripts/loop_cut_video.py shots/shot-002.mp4 --out shot-002-loop.mp4

Source code: https://github.com/kndlt/sprite-flux/blob/main/scripts/split_shots.py

Shot 1: Saying Hi

Shot 2: Idle Loop

Shot 3: Run Cycle

Post Processing: Detect Loop

Splitting is working fine. Let’s now look at extracting animation cycles using the strategy explored in previous post.

> poetry run python ./scripts/loop_cut_video.py shots/shot-002.webp --out shot-002-loop.webp

Loop: 18->30 len=12 seam=0, avg_mot=746.245, score=0.001
Saved shot-002-loop.webp
start:      18
end:        30
length:     12
seam:       0
avg_motion: 746.2448
score:      0.0006700213998556137
fps:        23.80952380952381

Source code: https://github.com/kndlt/sprite-flux/blob/main/scripts/loop_cut_video.py

We will ignore “saying hi” shot since that one is just a dummy one.

Final Outcome

Idle Loop

Run Loop

Conclusion

We now have series of python scripts to run to extract the animation clips. In next post, we will build custom nodes for ComfyUI.

Sprite DX - Stage 3 - Splitting Shots