Integrating Stage 1 into SpriteDX Web UI
Sprited Dev
So far, we’ve got:
RunPod Serverless running ComfyUI service in a headless mode.
Vite React SPA with simple API endpoint
/api/runthat proxies the requests to the RunPod.
We got all the pieces required to integrate Stage 1 through 4 of the SpriteDX pipeline.
We’ve also built out ComfyUI custom nodes in previous posts to allow for the full pipeline to run inside Comfy workflows.
Stage 1 Integration
Today, I will start integration of Stage 1 into WebUI. Here is screenshot of Stage 1 Workflow
Models
FLUX.1 Fill [dev]
It uses flux1-fill-dev.safetensors. There is a little bit of problem. Commercially using this model will require $999/mo (for up to 100,000 images $0.01 per generated image above 100,000).
It’s great for serving lots of users but not so great for current stage (we have 0 users 😇). So, our best alternative is to use FLUX.1 Fill [pro] model which is typically around $0.05 per image.
If you think about it though, 5 cent for generating character is not so bad. If you had to create one by hand, it would have taken one or two hours.
ae.safetensors, clip_l.safetensors, and t5xxl_fp16.safetensors
The ones that comes bundled with Flux.
Inputs
Template Character Sheet: We hand craft these character sheets and allow user to pick from a preset or upload their own.
Prompt: Helps guiding the generation. If you put "with glasses,” it generates characters with glasses on.
Masks: Masks are randomly generated and applied.
Outputs
- Images: The process generates 1~4 characters at once. So we will have those images returned.
Custom Nodes
So far, there wasn’t any custom nodes that needed to be ran. For FLUX.1 Fill API, we would use “Flux. 1 Fill Image” api node that comes packaged with ComfyUI.
Replacing local KSampler with API node
In this section, we replace the KSampler with a Black Forest Lab API node. Here is what we have currently.
We replace it with API node.
There is a problem though, the API node that comes pre-included uses a proxy server that goes from Client → Comfy → BFL. We need to purchase credits from Comfy if we want to use this node.
Account setup can be done in following website:
(little hard to find from comfy.org, just type in the url by yourself).
The UI is rather awful. There isn’t a button from comfy.org to platfom.comfy.org, and it is little confusing where to go if you are first time user at comfy.org
There’s no button to log in‼️ So you have to type in the url yourself.
Once you are in https://platform.comfy.org, you can log in but it only have 3 options — API Keys, Usage, and User.
There is no payment page‼️ Apparently, you currently have to download the Comfy and pay from there it seems like.
So, that’s not so reassuring. Let’s just use a different node that doesn’t use proxy service that Comfy API nodes use.
The official API custom nodes that BFL offers can be found here:
https://github.com/black-forest-labs/bfl-comfy-nodes
[~/dev/ComfyUI/custom_nodes]$ git clone git@github.com:black-forest-labs/bfl-comfy-nodes.git
Then, follow the instructions on the README to add environment variables.
You should nowbe able to use this node after a restart.
Okay, all good. Before we proceed to exporting this workflow let’s install the same BFL nodes into our headless version of Comfy that’s running in RunPod.
Customizing runpod-workers/worker-comfyui
Now, we build the Docker image with the BFL’s custom node installed. Here is the main documentation:
https://github.com/runpod-workers/worker-comfyui/blob/main/docs/customization.md
The base image is going to be runpod/worker-comfyui:5.4.0-base, and we simply add a RUN step to install the BFL custom node.
FROM runpod/worker-comfyui:5.4.0-base
# install custom nodes using comfy-cli
RUN git clone https://github.com/black-forest-labs/bfl-comfy-nodes.git /comfyui/custom_nodes/bfl-comfy-nodes
# Test input
COPY test_input.json /test_input.json
Here is my test_input:
{
"input": {
"workflow": {
"9": {
"inputs": {
"filename_prefix": "ComfyUI",
"images": [
"69",
0
]
},
"class_type": "SaveImage",
"_meta": {
"title": "Save Image"
}
},
"17": {
"inputs": {
"image": "template.png"
},
"class_type": "LoadImage",
"_meta": {
"title": "Load Image"
}
},
"46": {
"inputs": {
"image": "mask.png",
"channel": "red"
},
"class_type": "LoadImageMask",
"_meta": {
"title": "Load Image (as Mask)"
}
},
"47": {
"inputs": {
"width": 364,
"height": 364,
"x": 364,
"y": 0,
"image": [
"69",
0
]
},
"class_type": "ImageCrop",
"_meta": {
"title": "Image Crop"
}
},
"51": {
"inputs": {
"width": 364,
"height": 364,
"x": 1092,
"y": 0,
"image": [
"69",
0
]
},
"class_type": "ImageCrop",
"_meta": {
"title": "Image Crop"
}
},
"52": {
"inputs": {
"width": 364,
"height": 364,
"x": 0,
"y": 364,
"image": [
"69",
0
]
},
"class_type": "ImageCrop",
"_meta": {
"title": "Image Crop"
}
},
"53": {
"inputs": {
"width": 364,
"height": 364,
"x": 728,
"y": 364,
"image": [
"69",
0
]
},
"class_type": "ImageCrop",
"_meta": {
"title": "Image Crop"
}
},
"59": {
"inputs": {
"filename_prefix": "ComfyUI",
"images": [
"47",
0
]
},
"class_type": "SaveImage",
"_meta": {
"title": "Save Image"
}
},
"60": {
"inputs": {
"filename_prefix": "ComfyUI",
"images": [
"51",
0
]
},
"class_type": "SaveImage",
"_meta": {
"title": "Save Image"
}
},
"61": {
"inputs": {
"filename_prefix": "ComfyUI",
"images": [
"52",
0
]
},
"class_type": "SaveImage",
"_meta": {
"title": "Save Image"
}
},
"62": {
"inputs": {
"filename_prefix": "ComfyUI",
"images": [
"53",
0
]
},
"class_type": "SaveImage",
"_meta": {
"title": "Save Image"
}
},
"63": {
"inputs": {
"direction": "right",
"match_image_size": true,
"spacing_width": 0,
"spacing_color": "white",
"image1": [
"47",
0
],
"image2": [
"51",
0
]
},
"class_type": "ImageStitch",
"_meta": {
"title": "Image Stitch"
}
},
"64": {
"inputs": {
"direction": "right",
"match_image_size": true,
"spacing_width": 0,
"spacing_color": "white",
"image1": [
"52",
0
],
"image2": [
"53",
0
]
},
"class_type": "ImageStitch",
"_meta": {
"title": "Image Stitch"
}
},
"65": {
"inputs": {
"direction": "down",
"match_image_size": true,
"spacing_width": 0,
"spacing_color": "white",
"image1": [
"63",
0
],
"image2": [
"64",
0
]
},
"class_type": "ImageStitch",
"_meta": {
"title": "Image Stitch"
}
},
"66": {
"inputs": {
"filename_prefix": "ComfyUI",
"images": [
"65",
0
]
},
"class_type": "SaveImage",
"_meta": {
"title": "Save Image"
}
},
"69": {
"inputs": {
"prompt": "pixelated retro pixel art cute NPC characters\n\n",
"seed": 605837099,
"guidance": 60,
"steps": 50,
"prompt_upsampling": false,
"safety_tolerance": 2,
"api_key_override": "",
"region": "EU1",
"image": [
"17",
0
],
"mask": [
"46",
0
]
},
"class_type": "FLUX 1.0 [fill]",
"_meta": {
"title": "FLUX 1.0 [fill]"
}
}
},
"images": [
{
"name": "template.png",
"image": "data:image/png;base64,…"
},
{
"name": "mask.png",
"image": "data:image/png;base64,…"
}
]
}
}
We then build the docker image
docker build --platform linux/amd64 -t sprited/worker-comfyui:latest .
Then push to docker hub.
docker login
docker push sprited/worker-comfyui:latest
Then go to RunPod Serverless to use the published image.
Also, don’t forget to set the environment variable in RunPod.
BFL_API_KEY=___black_forest_labs_api_key___
Exporting workflow from ComfyUI
Now, let’s hop on over to ComfyUI again and export the workflow. ComfyUI > Workflow > Export (API).
Now, we should be able to supply this to our headless comfy in RunPod Serverless and generate characters.
Integration
To test integration, I linked up “Generate“ button to call the API and to store the output images into files section.
Result
Clicking on Generate button, calls the API and returns set of characters. 🎉
— Sprited Dev 🌱
Subscribe to my newsletter
Read articles from Sprited Dev directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by