Inception

Hello everyone, this article is part of The Azure Series, and it's not dependent on any previous articles. I use this series to publish-out Azure Projects & Knowledge.

Overview - Azure Python SDKs

Microsoft Azure produces Python SDK libraries that enable Developers to interact with existing Azure services or craft new ones; By Python code you run locally or in the cloud (i.e. function app - as Next Blog posts will explain)

Azure has Python SDK libraries composed of over 180 individual libraries. The Azure SDKs support Python 3.8 or later. Azure supports multiple ways to authenticate while working locally, which will discover earlier.

The open-source Azure libraries for Python simplify provisioning, managing, and using Azure resources from Python application code.[1]

The Azure libraries are how you communicate with Azure services from Python code that you run either locally or in the cloud.[1]

[1] Use the Azure libraries (SDK) for Python

Azure distinct its SDKs as ‘Management’ and ‘Client’ Libraries, sometimes referred to as ‘management plan’ and ‘data plan’

Consider this distinction as groups hold Azure SDKs, each set or group composed of a list of SDKs that serve a specific purpose.

Management Libraries overview

The SDK's management (or "management plane") libraries, the names of which all begin with azure-mgmt-, help you create, configure, and otherwise manage Azure resources from Python scripts. All Azure services have corresponding management libraries. For more information, see Azure control plane and data plane.

With the management libraries, you can write configuration and deployment scripts to perform the same tasks that you can through the Azure portal or the Azure CLI. (As noted earlier, the Azure CLI is written in Python and uses the management libraries to implement its various commands.)[2]

[2] Use the Azure libraries (SDK) for Python - Python on Azure | Microsoft Learn

Client Libraries overview

The SDK's client (or "data plane") libraries help you write Python application code to interact with already-provisioned services. Client libraries exist only for those services that support a client API.

The article, Example: Use Azure Storage, provides a basic illustration of using client library.[3]

[3] Use the Azure libraries (SDK) for Python - Python on Azure | Microsoft Learn

⚠

Reference: Azure Python SDK libraries documentation found here Reference | Microsoft Learn

Interesting details

Azure CLI is written in Python management libraries. Therefore, anything you can do with Azure CLI, you can achieve it as well with Python

the CLI commands provide many helpful features such as performing multiple tasks together, automatically handling asynchronous operations, formatting output like connection strings, and so on. So, using the CLI (or its equivalent, Azure PowerShell) for automated creation and management scripts can be more convenient than writing the equivalent Python code, unless you want to have a much more exacting degree of control over the process.[4]

[4] Use the Azure libraries (SDK) for Python - Python on Azure | Microsoft Learn

Authentication methods

There are multiple ways to authenticate locally, a chunk of them listed below:

AzureCliCredential: Let’s authenticate using command line interface by running az login
- First and foremost, install Azure CLI by following the steps here: Install the Azure CLI on Linux | Microsoft Learn
- Start Authenticate by running az login Command.
- Install Azure identity SDK by pip install azure.identity Command.
- Start typing a Python script for authentication as follows:

from azure.identity import *
credential = AzureCliCredential()  # Fetch CLI authentication

AzurePowerShellCredentials: same as AzureCliCredential. However, by using PowerShell command Connect-AzAccount
- Run Connect-AzAccount from your PowerShell Terminal.
- Install The Azure identity SDK by pip install azure.identity Command.
- Start typing a Python script for authentication as follows steps:

from azure.identity import *
credential = AzurePowerShellCredential()

VisualStudioCodeCredential: Authenticate using Azure extension
- by installing the extension below and authenticating.
- Then fetch authentication credentials as following code.
- First and foremost, install The Azure identity SDK by pip install azure.identity Command.

from azure.identity import *
credential = VisualStudioCodeCredential()

Note: This doesn’t work with newer versions of VS Code extention, so you should prefer AzureCliCredential() to authentication with VS Code.

Jose Portilla

EnvironmentCredential: Authenticate by pulling certain predefined variables.
- Start with installing The Azure identity SDK by pip install azure.identity Command, Then:

from azure.identity import *
credential = EnvironmentCredential()

# If you prefer storing your credentials as environment variables, 
# you can use `EnvironmentCredential()`. It can attempt to authenticate 
# with the following variables:

* `AZURE_TENANT_ID` - ID of the service principal's tenant
* `AZURE_CLIENT_ID` - Client ID for the service principal
* `AZURE_CLIENT_SECRET` - Client secret for the service principal
* `AZURE_AUTHORITY_HOST` - Authority of an Azure Active Directory endpoint
* `AZURE_CLIENT_CERTIFICATE_PATH` - Path to a PEM or PKCS12 certificate file, which includes the private key
* `AZURE_CLIENT_CERTIFICATE_PASSWORD` - Password for the certificate file
* `AZURE_USERNAME` - User name for Azure account/application
* `AZURE_PASSWORD` - Password for the Azure user

Optional - Store required environment variables in a file .env (i.e. consider ignoring pushing .env file with .gitignore)

              # .env file content
              AZURE_TENANT_ID="ID of the service principal's tenant"
              AZURE_CLIENT_ID="Client ID for the service principal"
              AZURE_CLIENT_SECRET="Client secret for the service principal"
              AZURE_AUTHORITY_HOST="Authority of an Azure Active Directory endpoint"
              AZURE_CLIENT_CERTIFICATE_PATH="Path to a PEM or PKCS12 certificate file, which includes the private key"
              AZURE_CLIENT_CERTIFICATE_PASSWORD="Password for the certificate file"
              AZURE_USERNAME="User name for Azure account/application"
              AZURE_PASSWORD="Password for the Azure user"

Optional - Then, fetch these variables by using setting.py file

              # setting.py file content
              import os
              from dotenv import load_dotenv

              # Get the path to the directory this file is in
              BASEDIR = os.path.abspath(os.path.dirname(__file__))
              # Load environment variables
              load_dotenv(os.path.join(BASEDIR, ".env"))
              AZURE_SUBSCRIPTION_ID = os.getenv("AZURE_SUBSCRIPTION_ID")
              DATA_LAKE_CONNECTION_STRING = os.getenv("DATA_LAKE_CONNECTION_STRING")
              CONNECTION_STRING = os.getenv("CONNECTION_STRING")
              AZURE_CLIENT_ID = os.getenv("AZURE_CLIENT_ID")
              AZURE_TENANT_ID = os.getenv("AZURE_TENANT_ID")
              AZURE_CLIENT_SECRET = os.getenv("AZURE_CLIENT_SECRET")
              STORAGE_ACCESS_KEY = os.getenv("STORAGE_ACCESS_KEY")

              # Default variables
              DEFAULT_RESOURCE_GROUP = "default-resource-group"
              DEFAULT_LOCATION = "eastus"

              # print(f"Using Azure Subscription ID: {AZURE_SUBSCRIPTION_ID}")

Optional - Then, import setting.py file into your main, here’s an example of entire main file content

              # example of main file content
              from azure.identity import *
              from settings import AZURE_SUBSCRIPTION_ID

              #credential = EnvironmentCredential()

              resource_client = ResourceManagementClient(credential, AZURE_SUBSCRIPTION_ID)

Optional - If you are looking for more info about EnvironmentCredential module use the below
```python from azure.identity import * import json

environment_cre_attr = dir(EnvironmentCredential()) print(json.dumps(environment_cre_attr, indent=4, default=str))

[ "class", "delattr", "dict", "dir", "doc", "enter", "eq", "exit", "format", "ge", "getattribute", "getstate", "gt", "hash", "init", "init_subclass", "le", "lt", "module", "ne", "new", "reduce", "reduce_ex", "repr", "setattr", "sizeof", "str", "subclasshook", "weakref", "_credential", "close", "get_token", "get_token_info" ]



    5. **InteractiveBrowserCredential:** by pop-up an a browser interactive session.


    * Start with installing Azure identity SDK by `pip install azure.identity` Command, Then:


```python
from azure.identity import *
credential = InteractiveBrowserCredential()

UsernamePasswordCredential: Authenticate by hard coding the user name and password
- Start with installing Azure identity SDK by pip install azure.identity Command, Then:

from azure.identity import *
credential = usernamePasswordCredential(
    Client_id: str,
    username: str,
    password: str,
    **kwargs,

)

DefaultAzureCredential: Attempts authentication by going through a list of other potential authentication methods until one succeeds. The authentication order is as follows:
- EnvironmentCredential.
- ManagedIdentityCredential.
- AzureCliCredential.
- AzurePowerShellCredential.
- InteractiveBrowserCredential.
- VisualStudioCodeCredential.
- UsernamePasswordCredential.

💡

For more details, check out Jose Portilla Jupyter notebook: Authentication Notebook

Management Libraries - Create and manage resources

The SDK's management (or "management plane") libraries, the names of which all begin with azure-mgmt-, help you create, configure, and otherwise manage Azure resources from Python scripts. All Azure services have corresponding management libraries. For more information, see Azure control plane and data plane.[5]

[5] Use the Azure libraries (SDK) for Python - Python on Azure | Microsoft Learn

While using Management Libraries “Control plane“ in the background you are accessing the Azure resource manager URLs, separated as follows:

For Azure global, the URL is https://management.azure.com.

For Azure Government, the URL is https://management.usgovcloudapi.net/.

For Azure Germany, the URL is https://management.microsoftazure.de/.

For Microsoft Azure operated by 21Vianet, the URL is https://management.chinacloudapi.cn. [6]

[6] Control plane and data plane operations - Azure Resource Manager | Microsoft Learn

Let’s discover The Mamangemet Libraries practically by creating a resource group

Craft requirements.txt file holds all the required Modules. Then install:

# requirements.txt content
azure.mgmt.resource
azure.identity
python-dotenv

Optional - Create a Python environment

python -m venv az-mgmnt-env

pip install -r requirements.txt

Create a .env file that holds the subscription ID, Avoiding to hard code it into your script

💡

.env file should be ignored to push to your repo

# .env file content
AZURE_SUBSCRIPTION_ID="PLACE_YOUR_SUB_ID_HERE"

Fetch .env file variables into setting.py script, as follows

💡

Usually, we should be using setting.py to handle environment variables and dependencies.

# setting.py content
import os
from dotenv import load_dotenv

# Get the path to the directory this file is in
BASEDIR = os.path.abspath(os.path.dirname(__file__))
# Load environment variables
load_dotenv(os.path.join(BASEDIR, ".env"))
AZURE_SUBSCRIPTION_ID = os.getenv("AZURE_SUBSCRIPTION_ID")
DATA_LAKE_CONNECTION_STRING = os.getenv("DATA_LAKE_CONNECTION_STRING")
CONNECTION_STRING = os.getenv("CONNECTION_STRING")
AZURE_CLIENT_ID = os.getenv("AZURE_CLIENT_ID")
AZURE_TENANT_ID = os.getenv("AZURE_TENANT_ID")
AZURE_CLIENT_SECRET = os.getenv("AZURE_CLIENT_SECRET")
STORAGE_ACCESS_KEY = os.getenv("STORAGE_ACCESS_KEY")

# Default variables
DEFAULT_RESOURCE_GROUP = "default-resource-group"
DEFAULT_LOCATION = "eastus"

# print(f"Using Azure Subscription ID: {AZURE_SUBSCRIPTION_ID}")

Authenticate using AzureCliCredential by:

az login

Create a main.py that holds the resource group creation, update, and deletion

"Module creates Azure Resource group"
from azure.identity import AzureCliCredential
from azure.mgmt.resource import ResourceManagementClient 
from settings import AZURE_SUBSCRIPTION_ID

# 1- Fetch Cli Credentials
credential = AzureCliCredential() 

# 2- Fetch resource management client - passing the required params value
resource_client = ResourceManagementClient(credential, AZURE_SUBSCRIPTION_ID)

# 3- Create a resouce group by using create_or_update method
rg_result = resource_client.resource_groups.create_or_update(
 'eraki_eastus_rg_1001',
 {'location': 'eastus'},   
)

print(rg_result.location)  # Print out resource group location
print(rg_result.name)  # Print out resource group name

# 4- Update resource group tags - as we are using create_or_update method, it provides us to update it
rg_upd_result = resource_client.resource_groups.create_or_update(
    resource_group_name='eraki_eastus_rg_1001',
    parameters={
        'location': 'eastus',
        'tags': {'environment': 'development', 'department': 'engineering'}
    }
)

print(rg_upd_result.tags)  # Print-out tags

# 5- List resource groups
group_list = resource_client.resource_groups.list()
for rg in group_list:
    print(f"Resource Group Name: {rg.name}, Location: {rg.location}, Tags: {rg.tags}")

# 6- Get resource group attributes
rg_info = resource_client.resource_groups.get('eraki_eastus_rg_1001')
print(rg_info)
print(rg_info.location)
print(rg_info.tags)

# 7- Delete resource group created
rg_result = resource_client.resource_groups.begin_delete(resource_group_name='eraki_eastus_rg_1001')
# "begin delete" which it takes a while to delete

# 8- List resource groups
rg_result.wait()  # wait until deletion finished
for rg in resource_client.resource_groups.list():
    print(f"Resource Group Name: {rg.name}, Location: {rg.location}")

Client Libraries - Manage resources

The SDK's client (or "data plane") libraries help you write Python application code to interact with already-provisioned services. Client libraries exist only for those services that support a client API.

The article, Example: Use Azure Storage, provides a basic illustration of using client library.[7]

[7] Use the Azure libraries (SDK) for Python - Python on Azure | Microsoft Learn

Let’s discover The Client Library “data plane“ practically by listing storage accounts & blob objects.

💡

First and foremost, you should have a Storage account blob placed out there to list from.

Create a Python environment

python -m venv az-client-env

List all dependencies SDKs in requiremts.txt, Then install

azure.identity
azure.storage.blob

pip install -r requirements.txt

Craft a .env file holds the required attributes, as follows:

# .env file content
AZURE_SUBSCRIPTION_ID="PLACE_SUB_ID_HERE"
ACCOIUNT_URL="https://STORAGE_ACCOUNT_NAME.blob.core.windows.net/"

💡

The .env shouldn’t be uploaded to git repos

Create a setting.py file that fetches the .env environments variables, as follows:

# setting.py content
import os
from dotenv import load_dotenv


def setting_func():
    # Get the path to the directory this file is in
    BASEDIR = os.path.abspath(os.path.dirname(__file__))
    # Load environment variables
    load_dotenv(os.path.join(BASEDIR, ".env"))
    AZURE_SUBSCRIPTION_ID = os.getenv("AZURE_SUBSCRIPTION_ID")
    DATA_LAKE_CONNECTION_STRING = os.getenv("DATA_LAKE_CONNECTION_STRING")
    CONNECTION_STRING = os.getenv("CONNECTION_STRING")
    AZURE_CLIENT_ID = os.getenv("AZURE_CLIENT_ID")
    AZURE_TENANT_ID = os.getenv("AZURE_TENANT_ID")
    AZURE_CLIENT_SECRET = os.getenv("AZURE_CLIENT_SECRET")
    STORAGE_ACCESS_KEY = os.getenv("STORAGE_ACCESS_KEY")
    ACCOIUNT_URL= os.getenv("ACCOIUNT_URL")

    # Default variables
    # DEFAULT_RESOURCE_GROUP = "default-resource-group"
    DEFAULT_RESOURCE_GROUP = "DefaultResourceGroup-NEU"
    DEFAULT_LOCATION = "eastus"

    # print(f"Using Azure Subscription ID: {AZURE_SUBSCRIPTION_ID}")

    return (
        AZURE_SUBSCRIPTION_ID,
        DATA_LAKE_CONNECTION_STRING,
        CONNECTION_STRING,
        AZURE_CLIENT_ID,
        AZURE_TENANT_ID,
        AZURE_CLIENT_SECRET,
        STORAGE_ACCESS_KEY,
        DEFAULT_RESOURCE_GROUP,
        DEFAULT_LOCATION,
        ACCOIUNT_URL,
    )

Craft a main.py script listing blob objects, by using AzureCliCredential

"Module listing Azure blob objects"    
from azure.identity import AzureCliCredential
from azure.mgmt.storage import StorageManagementClient
from azure.storage.blob import BlobServiceClient
from setting import setting_func

# return func values
(
    AZURE_SUBSCRIPTION_ID,
    DATA_LAKE_CONNECTION_STRING,
    CONNECTION_STRING,
    AZURE_CLIENT_ID,
    AZURE_TENANT_ID,
    AZURE_CLIENT_SECRET,
    STORAGE_ACCESS_KEY,
    DEFAULT_RESOURCE_GROUP,
    DEFAULT_LOCATION,
    ACCOIUNT_URL,
) = setting_func()

# Configure management client 
credential = AzureCliCredential()
storage_mgmt_client = StorageManagementClient(credential, AZURE_SUBSCRIPTION_ID)
# list storage accounts - by management client
for st in storage_mgmt_client.storage_accounts.list():
    print(f"{st.name}: {st.primary_endpoints.blob}")

# list blobs by BlobServiceClient
# with BlobServiceClient we able to configure storage account properties
blob_service_client = BlobServiceClient(
    account_url=ACCOIUNT_URL,
    credential=credential
)
container_client = blob_service_client.get_container_client(container="containerone")  
# Replace containerone with your contianer name
# ensure you set yourself Storage Blob Data Contributor role on the storage account, wait 3 mins to take affect.
for blob in container_client.list_blob_names():
    print(blob)

The output will be similar to this:

erakitest4001: https://erakitest4001.blob.core.windows.net/
azure_sdk_diagram.jpeg

💡

Ensure you set yourself to the “Storage Blob Data Contributor” role on the storage account for successful output. Wait 3 minutes for the change to take effect.

💡

Next Blob post will deploy our code into an Azure Function App.

Get Help

Encounter any issue!? Yeah, for sure you will, here are some resources:

Get help and connect with the SDK team

Visit the Azure libraries for Python documentation

Post questions to the community on Stack Overflow

Open issues against the SDK on GitHu b

Mention @AzureSDK on Twitter

Com plete a s hort survey about the Azure SDK for Python [8]

[8] Use the Azure libraries (SDK) for Python - Python on Azure | Microsoft Learn

Resources

That's it, Very straightforward, very fast🚀. Hope this article inspired you and will appreciate your feedback. Thank you

Azure Python SDK Libraries - Concept essentials

Table of contents