Cancelling Dataflow Gen2 Refresh

Update : Mar 8, 2024

As of March 8, you can cancel the Fabric Dataflow Gen2 refresh in service using UI:

Programmatically cancel the refresh:

DFg2 can be resource intensive, and if the refresh takes longer than expected, it may consume a significant amount of CUs. Thankfully, you can use the Power BI Rest API to cancel it programmatically. My friend Alex Powers already has a PowerShell script that you can use. You can also use the Power BI VS Code extension by Gerhard Brueckl.

But I would like to show you how you can do this using the PowerBIRestClient in the latest version of Semantic-Link (v0.5.0). The API is similar to the FabricRestClient I blogged about in my last blog. The new version has many updates I love and I will write about it soon. But in the meantime, checkout the official documentation for the details.

In the below code:

• I first get the list of all the dataflows in a workspace using the Get Dataflows API. You can pass either the workspace name or id. By default, the workspace id of the workspace hosting the notebook is used.

• For each dataflow, get the transaction id and the status of the dataflow using Get Dataflow Transactions API.

• Cancel the refresh, using the Cancel Dataflow Transaction API

Semantic Link makes calling APIs very easy. Authentication is handled automatically for you so no need to generate a token.

# Install semantic-link >= 0.5.0
!pip install semantic-link --q

import pandas as pd
import sempy.fabric as fabric
from sempy.fabric.exceptions import FabricHTTPException

# Instantiate the client
client = fabric.PowerBIRestClient()

def get_dataflow_transactions(workspace_id, dataflow_id):
    try:
        response = client.get(f"v1.0/myorg/groups/{workspace_id}/dataflows/{dataflow_id}/transactions")

        if response.status_code != 200:
            raise FabricHTTPException(response)
            return pd.DataFrame()

        transactions = response.json().get('value', [])
        if not transactions:
            return pd.DataFrame([{'status': 'No Status', 'transaction': 'No Transaction ID', 'refreshType': 'None'}])

        first_transaction = transactions[0]
        return pd.DataFrame([{
            'status': first_transaction['status'],
            'transaction': first_transaction['id'],
            'refreshType': first_transaction['refreshType']
        }])
    except Exception as e:
        print("Error getting transactions. Check workspace, dataflow id:", e)
        return pd.DataFrame()

def get_dataflows(workspace):
    try:
        workspace_id = fabric.resolve_workspace_id(workspace)
    except Exception as e:
        print("Check workspace name/id:", e)
        return pd.DataFrame()

    try:
        response = client.get(f"v1.0/myorg/groups/{workspace_id}/dataflows")
        if response.status_code != 200:
            raise FabricHTTPException(response)
            return pd.DataFrame()

        dataflows = response.json().get('value', [])
        dataflow_df = (pd.json_normalize(dataflows, max_level=1)
                       .rename(columns={"name": "dataflowName", "objectId": "dataflowID"})
                       .dropna()
                       .reset_index(drop=True))
        result = dataflow_df.join(dataflow_df.apply(lambda x: get_dataflow_transactions(workspace_id, x['dataflowID']).iloc[0], axis=1))
        return result
    except Exception as e:
        print("Error in get_dataflows:", e)
        return pd.DataFrame()


def cancel_dataflow(workspace, transaction_id):
    """
    Cancel a specific dataflow transaction.
    """
    try:
        workspace_id = fabric.resolve_workspace_id(workspace)
        response = client.post(f"v1.0/myorg/groups/{workspace_id}/dataflows/transactions/{transaction_id}/cancel")
        if response.status_code != 200:
            raise FabricHTTPException(response)
            return None

        return response.json()
    except Exception as e:
        print("Error:", e)
        return None

Example:

Get dataflow transactions:

Cancel the dataflow:

This was one example . You can use this client to call any Power BI Rest endpoint and make GET, POST requests. If you are new to Python, note that the actual code is just a couple of lines, majority is for error/exception handling to make it robust. I love Semantic Link, and it's becoming indispensable day by day.