Microsoft Fabric REST API's

Microsoft Fabric provides a rich set of API's that can be utilized to automate majority of the task required to maintain your Fabric tenant. With Fabric API's, you could automate the process to create/update/delete Notebooks, ML experiments, Semantic models, Warehouses, list SQL Endpoints and tons of other stuff.

The exhaustive list of the API's is available here : Microsoft Fabric REST APIs.

In this post we would see on ways to leverage Fabric API's to perform the following tasks:

• Create a Lakehouse

• Move files into the lakehouse as tables. Note that at the time of this writing there isn't any API available to perform external data load into a lakehouse.

• Create OneLake shortcut from an Azure ADLS2 location

• Create OneLake shortcut between two different lakehouses

The Setup

To get started you need to register the application at https://entra.microsoft.com

As Microsoft Entra uses MFA(Multi Factor Authentication) you might want to first set that up for your login to work . You can find more details here :

https://learn.microsoft.com/en-us/entra/identity/authentication/concept-mfa-howitworks

Once logged in, navigate to Applications >>App registrations >> New registration

and register an application

I have the named it as : Fabric API Calls

Once registered, the application would be assigned an Application ID, Object ID and tenant ID. We would only require the Application ID to be referenced in the code.

The Code

In a new Console Application add two Nuget packages : NewtonSoft and Microsoft Identity.

Next, create a connection to the ADLS2 storage for the shortcut.

Declare a bunch of static variables under Program class in the Console Application.

    private static string ClientId = "Cliend Id which we got from registring it in Entra";
    private static string AzureADLSConnection = "Connection Id from the above image marked in red";
    private static string Authority = "https://login.microsoftonline.com/organizations";
    private static string RedirectURI = "http://localhost";
    private static string[] scopes = new string[] { "https://api.fabric.microsoft.com/Workspace.ReadWrite.All https://api.fabric.microsoft.com/Item.ReadWrite.All https://api.fabric.microsoft.com/OneLake.ReadWrite.All" };
    private static string responsename = "";
    private static string workspaceName = "Workspace name";

I have declared a variable called scopes

private static string[] scopes = new string[] { "https://api.fabric.microsoft.com/Workspace.ReadWrite.All https://api.fabric.microsoft.com/Item.ReadWrite.All https://api.fabric.microsoft.com/OneLake.ReadWrite.All" };

Basically scopes defines the level of permissions that are required to perform certain operations across the fabric tenant.
For example : Workspace.ReadWrite.All,Item.ReadWrite.AllOneLake.ReadWrite.All, allows to read and write across a workspace, item and OneLake.

Next we create a new class called MyService with two generic methods that would perform GET and POST operations. These operations would use Bearer token authentication. You can read more about Bearer authentication here .

A special shoutout to this article.

https://www.aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/

My design of the class MyService revolving around the usage of HTTPClient was inspired by the above article

In the class MyService, as stated earlier has two generic methods for GET and POST. I have named them as GetAsync and PostAsync.

   public class MyService
   {

       private static readonly HttpClient client = new HttpClient();
       public HttpClient Client => client;
       public async Task<string> GetAsync(string url)
       {


           PublicClientApplicationBuilder PublicClientAppBuilder =
                   PublicClientApplicationBuilder.Create(ClientId)
                   .WithAuthority(Authority)
                   .WithRedirectUri(RedirectURI);

           IPublicClientApplication PublicClientApplication = PublicClientAppBuilder.Build();

           AuthenticationResult result = await PublicClientApplication.AcquireTokenInteractive(scopes)
                                                .ExecuteAsync()
                                               .ConfigureAwait(false);

           Client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", result.AccessToken);
           HttpResponseMessage response = await client.GetAsync(url);
           response.EnsureSuccessStatusCode();
           return await response.Content.ReadAsStringAsync();
       }

       public async Task<string> PostAsync(string url, HttpContent content)
       {

           PublicClientApplicationBuilder PublicClientAppBuilder =
                   PublicClientApplicationBuilder.Create(ClientId)
                   .WithAuthority(Authority)
                   .WithRedirectUri(RedirectURI);

           IPublicClientApplication PublicClientApplication = PublicClientAppBuilder.Build();

           AuthenticationResult result = await PublicClientApplication.AcquireTokenInteractive(scopes)
                                                .ExecuteAsync()
                                               .ConfigureAwait(false);

           client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", result.AccessToken);
           HttpResponseMessage response = await client.PostAsync(url, content);
           try
           {
               response.EnsureSuccessStatusCode();
               return await response.Content.ReadAsStringAsync();
           }
           catch
           {
               Console.WriteLine(response.Content.ReadAsStringAsync().Result);
               return null;
           }

       }
   }

Main method in the Console Application

static async Task Main(string[] args)
{
    MyService service = new MyService();
    string baseUrl = "https://api.fabric.microsoft.com/v1/workspaces";
    responsename = await service.GetAsync(baseUrl);
    JObject jsonObject = JObject.Parse(responsename);
    int i = 0; string workspaceId = "";
    for (i = 0; i < jsonObject["value"].Count(); i++)
    {
        if (jsonObject["value"][i]["displayName"].ToString() == workspaceName)
        {
            workspaceId = jsonObject["value"][1]["id"].ToString();
        }
    }
}

Couple of things to note . I have used Newtonsoft library to parse the response and to fetch the workspaceIdof the desired workspace I want to manage by checking the workspace name in the for loop.

The workspacename used above is a static variable declared earlier.

private static string workspaceName = "Workspace name";

Now execute the program. At the launch, it would prompt for acceptance of the required permissions based on the scopes that were declared.

Once accepted, it would ask for the Fabric tenant login to set the Bearer token. Login into it using your Fabric tenant credentials.

Once the login is authenticated and everything else works ok , you would see the following screen.

Please note the set up will not work if your workspace License mode is Pro.

Now lets check the methods to perform the lakehouse operations :

Create a new Lakehouse

    static async Task CreateLakeHouse(string workspaceId, string displayName)
    {

        string baseUrl = $"https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/lakehouses";
        MyService service = new MyService();
        var jsonData = new Dictionary<string, string>
              {
                  { "displayName", displayName },
                  { "type", "Lakehouse"}
};
        string jsonString = Newtonsoft.Json.JsonConvert.SerializeObject(jsonData);
        var content = new StringContent(jsonString, Encoding.UTF8, "application/json");
        responsename = await service.PostAsync(baseUrl, content);
        Console.WriteLine(responsename);

    }

Call to the above method :

await CreateLakeHouse(workspaceId, "Fabric_API_LakeHouse");

where Fabric_API_LakeHouse is the name of the lakehouse to be created.

Move Files into a Table in a Lakehouse

As stated earlier, at the time of this writing there are no API's to perform a direct data load into the lakehouse. Only option available is to copy the data from Files to Tables in a lakehouse.

The method would copy demo_data.csv from Files to Tables as a delta table.

static async Task LoadTable(string workspaceId, string Lakehouse, string relativePath, string tablename)
{

    MyService service = new MyService();

    string baseUrl = $"https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/lakehouses";
    responsename = await service.GetAsync(baseUrl);
    JObject jsonObject_lakhouse = JObject.Parse(responsename);
    var lakehouseid = (jsonObject_lakhouse["value"].FirstOrDefault(lh => lh["displayName"]?.ToString() == Lakehouse));
    lakehouseid = lakehouseid["id"];

    baseUrl = $"https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/lakehouses/{lakehouseid}/tables/{tablename}/load";

    var jsonData = new Dictionary<string, object>
    {
        { "relativePath", relativePath },
        { "pathType", "File" },
        { "mode", "Overwrite" },
        { "recursive", false },
        { "formatOptions", new Dictionary<string, object>
            {
                { "format", "Csv" },
                { "header", true },
                { "delimiter", "," }
            }
        }
    };

    string jsonString = Newtonsoft.Json.JsonConvert.SerializeObject(jsonData);
    var content = new StringContent(jsonString, Encoding.UTF8, "application/json");
    responsename = await service.PostAsync(baseUrl, content);
    Console.WriteLine(responsename);

}

Call to the above method :

   await LoadTable(workspaceId, "Fabric_API_LakeHouse", "Files/demo_data.csv", "demo_data");

Few things to note. I have used a a LINQ method to fetch the lakehouseid based on LakeHouseargument passed to the function.

var lakehouseid = (jsonObject_lakhouse["value"].FirstOrDefault(lh => lh["displayName"]?.ToString() == Lakehouse)); lakehouseid = lakehouseid["id"];

Also have to mention the format option of the file to be moved
{ "formatOptions", new Dictionary<string, object> { { "format", "Csv" }, { "header", true }, { "delimiter", "," } } }

Post execution you would see the demo_data.csv copied to Tables of the lakehouse.

Create a Azure ADLS2 Shortcut

    static async Task CreateADLS_ShortCut(string workspaceId, string LakehouseName, string shortCutPath, string shortCutName, string sourceLocation, string sourcesubPath, string connectionId)
    {
        MyService service = new MyService();


        string baseUrl = $"https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/items";

        responsename = await service.GetAsync(baseUrl);
        JObject jsonObject_lakhouse = JObject.Parse(responsename);
        var lakehouseid = jsonObject_lakhouse["value"].FirstOrDefault(lh => lh["displayName"]?.ToString() == LakehouseName && lh["type"]?.ToString() == "Lakehouse");
        lakehouseid = lakehouseid["id"];

        baseUrl = $"https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/items/{lakehouseid}/shortcuts";

        var jsonData = new Dictionary<string, object>
        {
            { "path",  shortCutPath },
            { "name", shortCutName },
            { "target", new Dictionary<string, object>
                {
            { "adlsGen2", new Dictionary<string, object>
                {
                    { "location", sourceLocation },
                    { "subpath", sourcesubPath },
                    { "connectionId", connectionId }
                }
            }
            }
            }

};

        string jsonString = Newtonsoft.Json.JsonConvert.SerializeObject(jsonData);
        var content = new StringContent(jsonString, Encoding.UTF8, "application/json");
        responsename = await service.PostAsync(baseUrl, content);

    }

Call to the above method :

 await CreateADLS_ShortCut(workspaceId, "Fabric_API_LakeHouse", "Files/ADLS_Shortcut", "Source", "https://<storage_account>.dfs.core.windows.net/", "/<storage_account>/source", AzureADLSConnection);

The method requires the path to the shortcut, name for the shortcut, the ADLS2 connectionid that we created earlier and the ADLS2 URI.

The method would create a shortcut at path Files/ADLS_Shortcut from the ADLS2 location <storage_account>/Source in Azure to the lakehouse Fabric_API_LakeHouse.

ADLS2 location

The Source folder has two subfolders Source_1 and Source_2 and a text file 1.txt.

As seen below, the shortcut was created for the ADLS2 location

Create a OneLake Shortcut

    static async Task CreateLakeHouse_ShortCut(string workspaceId, string SourceLakehouse, string DestinationLakehouse, string SourceLakehousePath, string DestinationLakehousePath, string shortCutName)
    {

        MyService service = new MyService();
        string baseUrl = $"https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/items";
        responsename = await service.GetAsync(baseUrl);
        JObject jsonObject_lakhouse = JObject.Parse(responsename);
        var lakehouseid_source = jsonObject_lakhouse["value"].FirstOrDefault(lh => lh["displayName"]?.ToString() == SourceLakehouse && lh["type"]?.ToString() == "Lakehouse");
        lakehouseid_source = lakehouseid_source["id"];

        var lakehouseid_destination = jsonObject_lakhouse["value"].FirstOrDefault(lh => lh["displayName"]?.ToString() == DestinationLakehouse && lh["type"]?.ToString() == "Lakehouse");
        lakehouseid_destination = lakehouseid_destination["id"];

        baseUrl = $"https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/items/{lakehouseid_destination}/shortcuts";

        var jsonData = new Dictionary<string, object>
        {
            { "path", SourceLakehousePath },//SourceLakehousePath
            { "name",shortCutName },//shortCutName
            { "target", new Dictionary<string, object>
                {
            { "oneLake", new Dictionary<string, object>
                {
                    { "workspaceId",  workspaceId },
                    { "itemId", lakehouseid_source },
                    { "path",  DestinationLakehousePath}//DestinationLakehousePath
                }
            }
            }
            }

};

        string jsonString = Newtonsoft.Json.JsonConvert.SerializeObject(jsonData);
        var content = new StringContent(jsonString, Encoding.UTF8, "application/json");
        responsename = await service.PostAsync(baseUrl, content);

    }

Call to the method

await CreateLakeHouse_ShortCut(workspaceId, "LakeHouse_1", "LakeHouse_2", "Files/Customers", "Files/Customers", "CustomerShortcut");

The method requires Source and Destination OneLake locations. In this case the method creates shortcut from LakeHouse_1 to LakeHouse_2 from Files/Customers under LakeHouse_1 to LakeHouse_2.

Closing Notes

Through this article I tried to provide insights into the possibilities of the Fabric REST API's.
Once you get a basic understanding of the underlying structure of the formats and the methods, automating the most crucial aspects of your Fabric tenant through the REST API's would be an easy possibility. Thank you for reading !!!