Variable Libraries, The Fabric Unsung Hero

In case you missed it, the Variable Library team over on the Fabric PG quietly snuck out some pretty big updates relatively recently that are actually quite massive. Before we dive into, it’s important to understand why these updates are so huge.

Deployments and CICD have been a real thorn in the side for most teams since launch, specifically with regards to data pipelines. If you’re not familiar, when you build a data pipeline and add an activity, such as a copy or a lookup activity, you have to configure a connection. This is done using a couple dropdown selections, which is trivial.

After getting the connections all sorted, you kickoff your job and everything works wonderfully. All good, right? Well, no, not really.

When you make the dropdown selection in the UI, the JSON of the pipeline is being updated with a hardcoded GUID of the selection made.

Eh, who care? It’s working now, we’re good to go. Right?

Insert our good friend, CICD

Assuming you’re following good development practices, you’re likely working in a feature branch or a development environment. So, what happens when you need to check that code in and promote your new pipeline to production? Remember, we now have a hardcoded GUID in the JSON of our pipeline.

Unlike some of the other Fabric items that will remain connected using named references (think notebook and lakehouse references), data pipelines do not maintain a link based on named reference. Said different, when you deploy your code to a new workspace, the GUID reference in your JSON will not change. If you deploy a pipeline to a new workspace and immediately trigger, it will reference the GUIDs from the originating workspace.

Now that we understand the issue, let’s talk about a solution.

Variable Library, Early Days

Variable Libraries have been in preview for quite a while, why did it take so long to give them the respect they deserve?! When VL’s first launched, they could only be used with data pipelines, and they did not support usability with activity connections. Yeah, you read that correctly. We could use them to swap other values in our pipelines, but we couldn’t use them to do the highest priority (personal opinion) function needed, let me hot swap an activity connection.

For that reason, we had solutions popping up like the fabric-cicd repo by one of the Microsoft internal engineering teams. fabric-cicd is a hugely helpful CICD kickstarter that solved the issue of changing connections by essentially doing a find/replace during deployment to replace GUIDs and other references.

Having seen the VL’s, I suspected it was only a matter of time before we had full support, so I extended the fabric-cicd repo to dynamically generate a parameter.yml file using a VL. This was obviously not the intended use case, but it allowed me to leverage the VL item in preparation (and hope) that we’d get connection integration at some point.

If you’re interested, I do have an open PR on fabric-cicd repo (I owe some updates to the docs, sorry for being slow) where you can view the code mentioned above. That said, I need to revisit to determine its viability going forward, but I suspect it still has a place in the toolbelt.

Added support for dynamic creation of parameter.yml file using integration with VariableLibrary artifact by Lucid-Will · Pull Request #264 · microsoft/fabric-cicd

Variable Library, All Grown Up

Fast forward a few months, and the current state of VL finally warranted some updates to the Lucid Data Platform codebase. I’m going to make a bit of an assumption that people have at least experimented with how to use a VL, so I won’t be going super deep on a “how-to”. If there’s interest in doing so, I’d be happy to put together another tutorial, but I’ll keep this one relatively high level.

Let’s take a quick look under the hood at how these little fellas can make your life a whole lot easier.

Setup:

Firstly, you’ll want to create a VL item in your workspace. If you do a bit of proper planning, your future self will thank you. E.g, instead of mashing everything into a single library, I try to be a bit more organized and spun up multiple libraries for logical grouping.

Here’s a glimpse into the mind of a slightly OCD, very ADHD person that probably got carried away due to excitement and how they structured their libraries:

And if you crack the door a bit more:

You can probably pickup on the logical grouping for the rest of them, so I’ll spare you the screenshots. But, keeping things isolated this way tends to reduce my time spent scrolling endlessly looking for that one thing, so it’s helped quite a bit.

Implementing in Data Pipelines

Circling back to the previous pipeline example, time to get rid of all those hardcoded references. If you crack open a data pipeline, you’ll find the Variable Library tab at the bottom of your canvas. Here’s where you’ll initialize the variables from your VL’s for usage in the pipeline.

Next step is to replace all the hardcoded references in your pipeline activities with variable library references for the juicy flexibility.

You drop them in the same way you would a parameter or a variable.

Now, if you check the pipeline JSON, the hardcoded references are gone and you’re ready to jam.

This Is Really, Really Cool

So, yeah, this is actually a super duper cool and very welcomed enhancement to the Fabric stack. It gets us closer to closing one of the biggest gaps (personal opinion) Fabric pipelines have had compared to legacy ADF/Synapse in that we were missing Linked Service items. It’s also a huge W with respect to CICD and how we manage connections and other references during deployment.

On a semi-related note, another feature that was quietly rolled out recently - Schedules are now included as part of the metadata being deployed. If you deploy a data pipeline with an active schedule, the schedule will go with it as part of deployment (and will deploy in the same state). So, after spending about ~30 minutes trying to figure out why jobs were duplicating and failing over top of each other, I realized the issue was related to doing a test deployment to a feature branch. The variable libraries were deployed in two workspaces, and the schedules deployed as active, so be careful with that, I guess?

Anyway, happy coding, Fabricators!

Oh, if you’re interested in a rock-star pickup to augment your dev team, check us out over here at Lucid BI and feel free to connect on LinkedIn for more cool Fabric stuff.