How to Prevent the Waterfall Effect in Data Fetching

In this article, we’ll dive into a common issue developers face when fetching data in Next.js or any asynchronous JavaScript environment: the waterfall effect. This happens when data fetching occurs sequentially, leading to slower response times. It’s important to recognize when sequential fetching is unavoidable and when you should fetch data in parallel to optimize performance.

We’ll explore how to avoid this mistake by using Promise.all and Promise.allSettled, along with code examples to illustrate the differences.

Understanding the Waterfall Effect

The waterfall effect occurs when one asynchronous operation waits for the completion of another before starting, leading to unnecessary delays. Here's a simple example of sequential fetching:

async function fetchProductData() {
    // Fetch product data
    const product = await fetch('/api/products/1');
    const productData = await product.json();

    // Fetch ratings after fetching product
    const ratings = await fetch(`/api/ratings/${productData.id}`);  
    const ratingsData = await ratings.json();

    return { productData, ratingsData };
}

What’s happening here?

• First, we wait for the product data to be fetched.

• Then, we use the product ID to fetch its associated ratings.

This sequential pattern means that the second request (ratings) won’t even begin until the first request (product) is complete. This is fine when the second request depends on the first, but it can lead to performance issues when the data fetched is independent.

When the Waterfall Effect Is Unavoidable

In some cases, sequential fetching is necessary. For example, if you need the product ID from the first request to fetch related ratings, the waterfall effect is unavoidable. Here, it’s reasonable to accept a slight delay because the second request can’t proceed without the data from the first.

However, if you have independent data to fetch, you should always fetch in parallel. Let’s look at how to optimize performance in these cases.

Optimizing with Parallel Data Fetching

For independent data fetching (when requests don’t rely on each other), we can use Promise.all or Promise.allSettled to perform parallel data fetching and eliminate unnecessary delays.

Using Promise.all

async function fetchData() {
    const [productsResponse, ratingsResponse] = await Promise.all([
        fetch('/api/products'),
        fetch('/api/ratings')
    ]);

    const productsData = await productsResponse.json();
    const ratingsData = await ratingsResponse.json();

    return { productsData, ratingsData };
}

Explanation:

• Promise.all initiates both fetch requests simultaneously.

• The responses are awaited in parallel, meaning both the product and ratings data are fetched concurrently, cutting down on total load time.

Caution: If any of the promises in Promise.all fails, the entire operation rejects, and none of the data will be returned. This could be problematic if one endpoint is unreliable.

Using Promise.allSettled

In cases where you expect some requests to fail but still want the others to succeed, use Promise.allSettled.

async function fetchData() {
    const results = await Promise.allSettled([
        fetch('/api/products'),
        fetch('/api/ratings')
    ]);

    const productsData = results[0].status === 'fulfilled' ? await results[0].value.json() : null;
    const ratingsData = results[1].status === 'fulfilled' ? await results[1].value.json() : null;

    return { productsData, ratingsData };
}

Explanation:

• Promise.allSettled ensures that all promises resolve, even if one fails.

• Each result contains a status field (fulfilled or rejected), allowing you to handle success and failure independently.

This approach is useful if you want to ensure that even if one of the API calls fails, the other requests still complete successfully.

Key Differences Between Promise.all and Promise.allSettled

• Promise.all: If one request fails, the entire operation fails.

• Promise.allSettled: All requests resolve, regardless of whether some fail.

When Should You Use Sequential Fetching?

Sequential fetching is only appropriate when there’s a dependency between the requests. For instance, if you need product details first to fetch its ratings, sequential fetching makes sense. But if the data is independent (like fetching product categories and user reviews), then parallel fetching is more efficient.

Example: Mixing Sequential and Parallel Fetching

Let’s say you need to fetch product data and ratings (dependent on the product ID), but also need to fetch independent data like user reviews. You can mix sequential and parallel fetching for an optimized solution:

async function fetchAllData() {
    // Fetch products and reviews in parallel
    const [productResponse, reviewsResponse] = await Promise.all([
        fetch('/api/products/1'),
        fetch('/api/reviews')
    ]);

    const productData = await productResponse.json();
    const reviewsData = await reviewsResponse.json();

    // Fetch ratings after getting product data
    const ratingsResponse = await fetch(`/api/ratings/${productData.id}`);
    const ratingsData = await ratingsResponse.json();

    return { productData, ratingsData, reviewsData };
}

What happens here?

• Product data and reviews are fetched in parallel because they are independent of each other.

• Once the product data is available, we use its ID to fetch the ratings in a sequential manner (since ratings depend on the product).

This approach strikes a balance, ensuring that independent data is fetched quickly while handling dependent requests in sequence when necessary.

Conclusion

The waterfall effect in data fetching can lead to slower performance, but it’s not always a bad thing. It’s important to distinguish between cases where sequential fetching is necessary (e.g., dependent data) and where parallel fetching can improve performance.

By using Promise.all for parallel requests and Promise.allSettled when some requests might fail, you can avoid unnecessary delays and improve the overall efficiency of your application.

Key Takeaways:

• Avoid the waterfall effect when fetching independent data by using parallel fetching (e.g., Promise.all).

• For dependent data (where one fetch relies on the result of another), sequential fetching is necessary.

• Use Promise.allSettled when you expect some requests to fail but want others to succeed.

By being mindful of these patterns, you can ensure your data fetching is optimized, leading to faster and more efficient applications!