How to Create Images from Text Using Stable Diffusion and Python

The ability to generate high-quality images from text descriptions has become a fascinating application of AI, especially with the advent of powerful models like Stable Diffusion. In this blog, we will explore how to create images from text using Stable Diffusion in Python, leveraging a model pre-trained by Hugging Face's Diffusers library and the Stable Diffusion Pipeline. This process enables generating detailed images based solely on textual prompts, which can be useful for artists, content creators, or even just for experimentation.

In this guide, you will learn how to:

• Set up the environment.

• Load a pre-trained Stable Diffusion model.

• Generate images from text prompts.

• Save the generated images for further use.

What is Stable Diffusion?

Stable Diffusion is a latent text-to-image diffusion model that generates realistic images based on textual descriptions. It was developed by CompVis, and its first version (v1.4) is one of the most commonly used models in various AI applications. Stable Diffusion is highly efficient in creating high-quality and detailed images from relatively simple text inputs, offering a range of creative possibilities.

Step 1: Setting up the Environment

Before diving into the code, ensure that you have all the necessary dependencies installed. You will need:

• PyTorch: For utilizing GPU acceleration (if available).

• Diffusers: The Hugging Face library for Stable Diffusion models.

• Transformers and Accelerate: To handle model optimization and processing.

You can install all the required libraries using pip:

pip install torch diffusers transformers accelerate

Install CUDA (Optional, but recommended)

If your machine has an NVIDIA GPU, installing CUDA can greatly accelerate the image generation process. If you're using a CPU, the process will still work but much slower.

For installing CUDA, follow the official PyTorch installation guide.

Code Breakdown

import torch
from diffusers import StableDiffusionPipeline

# Check if CUDA is available and use it if possible
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using device: {device}")

# Load the pre-trained Stable Diffusion model
pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
pipe.to(device)

Explanation:

• torch.cuda.is_available() checks whether a GPU is available and assigns the device accordingly (cuda for GPU, cpu otherwise). Using a GPU can make image generation significantly faster.

• StableDiffusionPipeline.from_pretrained() loads the pre-trained Stable Diffusion model from Hugging Face's model repository.

• pipe.to(device) ensures the model is transferred to the specified device (either GPU or CPU).

This setup ensures that we have the Stable Diffusion model ready for generating images.

Step 3: Generating Images from Text

Now that the model is loaded, let's write a function that takes a text prompt as input and generates an image based on that prompt.

def generate_image_from_text(text_prompt):
    image = pipe(text_prompt).images[0]
    return image

How it works:

• The pipe(text_prompt) command takes in the text description, runs it through the model, and returns a list of generated images. Since we're only generating one image, we select the first image with .images[0].

• The function returns the generated image object, which can then be manipulated or saved.

Step 4: Example of Image Generation

Now that we have everything in place, let’s try to generate an image from a simple text prompt.

# Example usage
text_prompt = "A beautiful sunrise over the mountains"
image = generate_image_from_text(text_prompt)
image.save('generated_image.png')

Here’s a breakdown:

• The text prompt is set to "A beautiful sunrise over the mountains". You can modify this to anything you’d like to visualize.

• The generated image is saved as generated_image.png in the current working directory.

Step 5: Visualizing the Generated Image

After saving the image, you can display it directly using Python’s built-in libraries such as Pillow or matplotlib.

from PIL import Image

# Open and display the generated image
image = Image.open('generated_image.png')
image.show()

Or, if you prefer using matplotlib:

import matplotlib.pyplot as plt

# Display the generated image using matplotlib
plt.imshow(image)
plt.axis('off')  # Hide the axis
plt.show()

This code will display the generated image in a new window or within your notebook, depending on where you run the code.

Step 6: Experimenting with Text Prompts

One of the most exciting aspects of text-to-image generation is the freedom to experiment with different textual prompts. Here are a few examples to try:

1. "A futuristic cityscape with flying cars"

2. "A cat wearing a space helmet in a spaceship"

3. "A fantasy castle on top of a hill surrounded by clouds"

4. "A dragon flying over a burning village"

5. "A peaceful garden with colorful flowers and a small pond"

Each prompt will generate a unique image, and the quality will vary based on the complexity of the prompt and the model’s ability to interpret it.

What Can You Do with Text-to-Image Generation?

With the ability to create stunning images from simple text descriptions, Stable Diffusion opens up numerous possibilities across various domains:

• Artistic Creation: Artists can use this technology to explore new ideas and visualize their concepts.

• Marketing and Content Creation: Content creators can generate unique visuals based on trending topics or specific themes.

• Prototyping and Design: Designers can create prototypes of environments, characters, or products without spending hours on manual creation.

• AI-Powered Tools: You can build tools for generating visual content automatically from descriptions or user inputs.

Resources:

• Hugging Face Diffusers Documentation

• Stable Diffusion GitHub Repository

This blog gives you a solid foundation for generating images from text. You can extend this further by fine-tuning the model on specific datasets, exploring more complex prompts, or combining the generated images with other AI models.