Service Principal for Microsoft Fabric Data Pipeline

Service Principal is supported in Microsoft Fabric as a more robust and reliable means of authentication. I had blogged about it some time ago where I demonstrated how Service Principal can be leveraged to query Fabric API’s. You can read it here.

Also there was another blog on using Service Principals in Azure.

In this post, we will explore how to leverage Service Principals in Microsoft Fabric, following a similar approach to Azure.

With Service Principal there is almost no operational overhead in terms of managing secrets, rotating credentials and ensuring secure access.

Fabric Workspace Identity

First we have to create a workspace identity specific for the workspace.

Creating a workspace identity automatically creates a managed service principal in your Entra account associated with that workspace.

To create a Workspace identity in Fabric , goto the Workspace Settings and create a new Workspace Identity.

Once done you should see a service principal created in your Entra account. It might not be immediate. Mine took around a minute.

SetUp

The ADLS2 source location has three csv files

In your Fabric tenant create a new connection

to the ADLS2 location.

and then select Authentication Type as Service Principal

In your Entra account open the Service principal created by the Workspace Identity.

Note : The Service Principal ID, Tenant ID, Service Principal Key in the connection settings of Fabric corresponds to the Application(Client)ID, Directory (tenant)ID and Client secret respectively in your workspace service principal account in Entra.

Ensure that you have granted at least the Contributor Access to the ADLS2 location

To test it ,create a new data pipeline and select the Copy Data assistant

and in the Copy Data Wizard select One Lake and you should see the new connection that was created. The one I created is named ADLS2.

Select it and if everything is set up correctly the connection should succeed.

What about shortcuts ?

You can create shortcuts in DataLake through this connection and it should work fine.

Advantages Of Service Principal ?

You might ask what advantages using a Service Principal has over the Access Key approach for ADSL2 ?

Well they both do provide the advantages of centralized credential management without the overhead of maintaining credentials separately , but consider a scenario where you require additional access for other resources on the tenant. With service principal you have to just grant access to the resource which is not a viable option with the Access Key approach. With Service Principal, permissions can be scoped to individual resources, resource groups or subscriptions.

Apart from this you can use a single service principal to authenticate multiple storage locations which isn't the case with access keys as every individual storage location would have their own set of access keys.With service principal, all you have to do is grant relevant access to the storage locations.

Also service principal are part of Entra the activities are fully auditable through AD logs at a granular level.

Conclusion

Service Principal in Fabric workspaces is a very secure way to authenticate the underlying data sources without the overhead of needing to store credentials across separate set of connections and it should be the most ideal way to implement external source authentications.

Thanks for reading !!!