Managing images in Azure can be both exciting and sometimes exhausting due to the variety of methods available. There are several approaches you can take, each with its own set of complexities and benefits. One common method is deploying your workloads directly from the Azure Marketplace, which offers a wide range of pre-configured virtual machine images. This approach is convenient and can save time, but it may not always meet your specific needs.

Alternatively, you can manually maintain your Golden VMs. This involves creating a base image that includes all the necessary configurations, software, and updates. Maintaining Golden VMs requires a more hands-on approach, as you need to ensure that the images are kept up-to-date and secure. Most of the time this will be done manually on a schedule. This method provides greater control and customization, allowing you to tailor the VMs to your exact requirements without the need to invest heavily in developing automations.

The solution I will discuss in this blog involves building and maintaining your images using image management tools like Azure Image Builder and HashiCorp Packer. Using these tools, you can streamline the process of image creation and maintenance, reducing the manual effort required and minimizing the risk of configuration drift. This approach not only saves time but also enhances the security and reliability of your virtual machines by ensuring they are built according to best practices and kept up to date with the latest updates and patches.

➡️ I use solutions like this in my own dev/demo environments to keep my images up to date, ensuring I always have the latest image available for demos and workshops.

A look into the future

I am planning to create a series of blog posts based on this topic and will be continuously developing the example shown in this blog. In these upcoming posts, I will dive deeper into each method, providing step-by-step guides, best practices, and real-world scenarios. My goal is to offer a comprehensive resource that can help you navigate the complexities of managing images in Azure. So, wish me luck 😅.

Another foreword

I will try to keep things simple in this series. In an enterprise environment, a setup like this can become quite complex due to limitations, connectivity restrictions, and complex requirements for privilege management and much more.

However, my primary goal with this series is to ensure that the examples remain clear and understandable. I want to avoid overengineering the examples so that they are accessible to everyone, regardless of their level of expertise. By focusing on simplicity, I hope to provide a solid foundation that you can build upon as you become more comfortable with the concepts.

That said, we might explore some of these more advanced topics in future posts.

Introduction

Before we start a few words about the tool we will use. HashiCorp Packer is an open-source tool that simplifies the creation of virtual machine images for Azure and other platforms from a single configuration. It automates the process of building custom virtual machine images, ensuring consistency and reducing errors. Packer's version-controllable configurations and integration with DevOps tools make it invaluable for efficient image management in Azure environments. By leveraging Packer, users can significantly improve their deployment workflows, saving time and enhancing reliability across different Azure setups.

Packer follows these steps to create a new image in Azure (high-level overview):

1. Validates the provided template

2. Create a temporary staging resource group

3. Deploy the VM template

4. Execute the defined build steps

5. Power off and capture the VM

6. Create and distribute the image

7. Delete the temporary resources

➡️ You'll find more details here: Azure Builder | Integrations | Packer | HashiCorp Developer

HashiCorp Packer vs Azure Image Builder

I won't go into too much detail about this topic here, as I plan to write a dedicated blog post on it. I have used both tools several times and really like them. Generally, I appreciate the simplicity of Packer, if that makes sense to you. Azure Image Builder adds an extra layer on top of Packer, offering more functionality and tighter integration with Azure.

Packer is a standalone, multi-cloud tool that provides a consistent workflow for creating images across various platforms, while Azure Image Builder is specifically designed for Azure. Azure Image Builder simplifies the image creation process within the Azure ecosystem, offering features like integration with managed identities, whereas Packer gives you more granular control and flexibility, especially when working across multiple cloud providers.

The teams are constantly improving the capabilities of Azure Image Builder, such as automatic runs, triggers, and built-in versioning logic for distribution.

➡️ More information: What's new in Azure VM Image Builder - Azure Virtual Machines | Microsoft Learn

Preparation

I recommend the following tools for developing and executing Packer templates on Azure. Some of them are optional, the Packer CLI is not.

• Packer CLI (Install Packer | Packer | HashiCorp Developer)

• PowerShell 7

• Azure CLI

• Visual Studio Code

• Visual Studio Code extension:

  • HashiCorp HCL - Visual Studio Marketplace

  • PowerShell - Visual Studio Marketplace

There is no true auto-completion feature for Packer in Visual Studio Code, but by installing the HashiCorp HCL extension, you can at least get syntax highlighting.

The final structure will look like something like this.

└── 📁az-image-packer-demo
    └── 📁.devcontainer
    └── 📁.github
    └── README.md
    └── 📁src
        └── az-windows-11-ent.pkr.hcl
        └── 📁scripts
            └── deprovisioning.ps1
            └── install-pwsh.ps1
            └── install-winget.ps1

Getting started

First things first. Before using Packer, we need to create a service principal to grant the necessary permissions to the subscription we plan to use for our deployment. By running the command az ad sp create-for-rbac --name 'packer-sp-name' --role Contributor --scopes /subscriptions/<subscription-id> we create a new service principal and grant it permission to the defined subscription. The output of this command can be used to setup the authentication inside our Packer template later.

In my case, I'm developing this on GitHub Code spaces, so I define the required properties as environment variables in my Code space. We will do something similar when we look at pipelines in some of the later blog posts.

Now let's create a new Azure Resource group using az group create --location eastus --name rg-p1-corp-packer-eus. That resource group will be used by Packer to create the managed image.

Creating a template

Packer templates can be written using JSON or HCL language. I prefer using HCL, so I will go with that. Let's run touch az-windows-11-ent.pkr.hcl to create our first template inside the ./src directory and open it in VS Code.

As mentioned earlier, we will build a very simple version of a Packer template with minimal requirements and dependencies on existing infrastructure. We will expand this template in the future to cover more advanced topics.

A Packer template consists of different sections that we will explore now. The full version of the template was uploaded here: blog-az-image-packer/blog/01-packer-getting-started/az-windows-11-ent.pkr.hcl at main · lrottach/blog-az-image-packer (github.com)

Variables

First, let's define some variables. You can use variables to make your template more flexible by parsing some values inside the template during execution. In my case, I am using the env() method to get the environment variables I defined earlier for authenticating Packer. Later we will use these variables to inject some more values into our template.

// Variables
// **********************

variable "client_id" {
  type    = string
  default = "${env("ENV_PACKER_APP_ID")}"
}

variable "client_secret" {
  type    = string
  default = "${env("ENV_PACKER_APP_SECRET")}"
}

variable "subscription_id" {
  type    = string
  default = "${env("ENV_PACKER_SUBSCRIPTION_ID")}"
}

variable "tenant_id" {
  type    = string
  default = "${env("ENV_PACKER_TENANT_ID")}"
}

Plugins

The great thing about Packer is its multi-platform capabilities. You can use Packer to manage your images on multiple clouds and platforms. In our case, we add the required plugin for Azure.

// Plugins
// **********************

packer {
  required_plugins {
    azure = {
      source  = "github.com/hashicorp/azure"
      version = "~> 2"
    }
  }
}

Source

Now it's time to configure our provider. Here, we define all the relevant properties for our image deployment like source, staging and destination. In this case, we will use and configure the azure-arm builder.
To get started, we configure the source block like this.

➡️ More information: Azure Builder | Integrations | Packer | HashiCorp Developer

source "azure-arm" "win-11-ent" {

  // Authentication
  client_id       = "${var.client_id}"
  client_secret   = "${var.client_secret}"
  subscription_id = "${var.subscription_id}"
  tenant_id       = "${var.tenant_id}"

  // Source image information
  os_type         = "Windows"
  image_offer     = "windows-11"
  image_publisher = "microsoftwindowsdesktop"
  image_sku       = "win11-23h2-ent"

  // Build VM size
  vm_size = "Standard_B8as_v2"

  // Location and public IP
  location      = "East US"
  public_ip_sku = "Standard"

  // Communicator (winrm) configuration
  communicator   = "winrm"
  winrm_use_ssl  = true
  winrm_insecure = true
  winrm_timeout  = "15m"
  winrm_username = "packer"

  // Managed Image information
  managed_image_resource_group_name = "<rg-name>"
  managed_image_name                = "windows11-ent-packer-image-v4-eus"
}

Explanation
In this block, we have configured all the necessary information for our image. I decided to start with a fresh image from the Azure Marketplace. Additionally, we included details about the build VM, such as its size, location, and public IP.

This is the simplest setup I can think of. Packer will handle most of the work for us. Apart from the image resource group, we don't have any prerequisites like networking, existing images and so on.

In this example, I will use Windows 11 Enterprise because I plan to use this image for Windows 365 and Azure DevBox, which do not support multi-session. Of course, you can create the same setup using Windows 11 Enterprise Multi-Session for Azure Virtual Desktop.

Overview

By applying this configuration, Packer will create a new build VM and some necessary resources like Azure KeyVault, public IP, and more. Packer will randomly generate the names for the staging resource group and all other resources.

In a later post in this series, we will look at how to gain more control over the temporary resources generated for staging, ensuring they comply with naming or tagging conventions in more restrictive environments.

Output

A successful run will create a new Azure Managed Image within the specified resource group.

Build

Lastly, we need to define the build block of our template. Here, we specify what actions Packer should apply to our image after the staging VM is created and Packer successfully connects using WinRM.

To keep things simple in this blog, we will install some apps. What is the best way to install and update applications without the hassle of downloading and automating setups? That's right, a package manager. 🦾

➡️ More information about provisioners: Provisioners | Packer | HashiCorp Developer

build {
  sources = ["source.azure-arm.win-11-ent"]

  provisioner "powershell" {
    inline = [
      # Installing chocolately package manager
      "Set-ExecutionPolicy Bypass -Scope Process -Force",
      "[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072",
      "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"
    ]
  }

  provisioner "windows-restart" {
    restart_check_command = "powershell -command \"& {Write-Output 'restarted.'}\""
  }

  provisioner "powershell" {
    inline = [
      # Get chocolately version
      "choco --version",
      # Install - Azure CLI
      "choco install azure-cli --confirm --silent",

      # Install - Git
      "choco install git --confirm --silent",

      # Install - PowerShell
      "choco install powershell-core --confirm --silent"
    ]
  }

  provisioner "powershell" {
    script = "./scripts/deprovisioning.ps1"
  }
}

Packer offers so called provisioners to run specific actions during staging. In this case, we will use the powershell provisioner to run scripts and inline commands. After installing the Chocolatey package manager, we perform a single reboot using the windows-restart provisioner before we install some apps.

Deprovisioning

In the final step, we need to deprovision our staging VM to create a fully functional image. The linked script starts the Sysprep process and waits until it is completed.

I recommend creating a script directory inside your project. Check out the linked GitHub repo to get an overview of the entire project and its structure.

Azure Image Builder is using a similar script like this, which is already built in.

# Ensure Guest Agent services are running
$guestAgentServices = @("RdAgent", "WindowsAzureTelemetryService", "WindowsAzureGuestAgent")
foreach ($service in $guestAgentServices) {
    $svc = Get-Service -Name $service -ErrorAction SilentlyContinue
    if ($svc) {
        while ((Get-Service $service).Status -ne 'Running') {
            Write-Output "Waiting for $service to start..."
            Start-Sleep -Seconds 5
        }
        Write-Output "$service is now running."
    }
    else {
        Write-Output "$service not found, skipping."
    }
}

# Run Sysprep
Write-Output "Starting Sysprep process..."
$sysprepPath = "$env:SystemRoot\System32\Sysprep\Sysprep.exe"
$sysprepArgs = "/oobe /generalize /quiet /quit /mode:vm"
Start-Process -FilePath $sysprepPath -ArgumentList $sysprepArgs

# Wait for Sysprep to complete
Write-Output "Waiting for Sysprep to complete..."
while($true) {
    $imageState = Get-ItemProperty HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Setup\State | Select-Object -ExpandProperty ImageState
    if($imageState -eq 'IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE') {
        Write-Output "Current state: $imageState"
        Write-Output "Sysprep completed successfully."
        Start-Sleep -Seconds 30
        break
    } else {
        Write-Output "Current state: $imageState"
        Start-Sleep -Seconds 10
    }
}

Write-Output "Deprovisioning process completed."

Why no Winget?

Talking about package managers on Windows, Winget is a great tool that I love using for setting up my Windows machines. However, when it comes to building images, Winget has some limitations and doesn't work right out of the box.

I spent hours (way too much 😅) on investigating this. In the end I managed to get a working Packer template which uses Winget for app installation. Unfortunately, this solution remains an odd workaround for now. It's probably fine for my own demo / dev environments, but not as a production solution. If you have further insight, please let me know.

The general issue is that Winget is not available as a command when accessing the VM during the Packer build phase. However, if you access the same virtual machine using Remote Desktop with a full desktop experience, Winget is available right away. This suggests that the problem is related to the method of connecting to the virtual machine. I don't want to spend too much time writing about this, but here is a brief summary of what I discovered.

The core issue appears to be related to how Winget interacts with the Windows system when run in a remote session. Specifically:

1. API Access: Winget requires access to certain Windows APIs that are not available or accessible in the context of a WinRM session. This limitation is likely due to the restricted environment that WinRM sessions operate in for security reasons.

2. User Context: There are indications that Winget may need to run in a specific user context, which is not fully replicated in a WinRM session. This could affect operations that require user-specific settings or permissions.

3. Shell Restrictions: WinRM sessions typically use a constrained version of PowerShell, which may not provide all the necessary capabilities that Winget relies on for its full functionality.

• I am having trouble using winget with packer. · microsoft/winget-cli · Discussion #3326 (github.com)

• Doesn't run under WinRM · Issue #256 · microsoft/winget-cli (github.com)

Inside my project on GitHub, you will find the scripts I used to install Winget during the build process. Be aware that using this script will cause Sysprep to fail at the end, because Winget is installed for the current WinRM user account. Therefore, you need to remove the related packages before running Sysprep. I also tried to rehost the Packer PowerShell builder to pwsh.exe instead of powershell.exe, but that didn't solve the problem.

Image build

Now it's time to test our previous work. For better readability, I highly recommend running the packer fmt ./az-windows-11-ent.pkr.hcl command from time to time to format your template correctly.

To install the Azure plugin for Packer that we configured in our template, we need to run the packer init ./az-windows-11-ent.pkr.hcl command. Without this, the build will fail.

I also recommend running the packer validate ./az-windows-11-ent.pkr.hcl command to test your Packer template without needing to do a full run.

To let the magic happen we can finally execute the packer build ./az-windows-11-ent.pkr.hcl command to create our image. In my case, the full build took about ten minutes.

I really like the live feedback you get from the Packer console. It's very easy to monitor the image build process and step in if there are any errors. Using Azure Image Builder, the console output does not exist like that. If there are errors, you need to download the log files from an Azure Storage Account.

The last parts of the build should look like this.

==> azure-arm.win-11-ent: Provisioning with Powershell...
==> azure-arm.win-11-ent: Provisioning with powershell script: ./scripts/deprovisioning.ps1
    azure-arm.win-11-ent: RdAgent is now running.
    azure-arm.win-11-ent: WindowsAzureTelemetryService not found, skipping.
    azure-arm.win-11-ent: WindowsAzureGuestAgent is now running.
    azure-arm.win-11-ent: Starting Sysprep process...
    azure-arm.win-11-ent: Waiting for Sysprep to complete...
    azure-arm.win-11-ent: Current state: IMAGE_STATE_COMPLETE
    azure-arm.win-11-ent: Current state: IMAGE_STATE_UNDEPLOYABLE
    azure-arm.win-11-ent: Current state: IMAGE_STATE_UNDEPLOYABLE
    azure-arm.win-11-ent: Current state: IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE
    azure-arm.win-11-ent: Sysprep completed successfully.
    azure-arm.win-11-ent: Deprovisioning process completed.
==> azure-arm.win-11-ent: Querying the machine's properties ...
==> azure-arm.win-11-ent:  -> ResourceGroupName : 'pkr-Resource-Group-xemqrr7bi4'
==> azure-arm.win-11-ent:  -> ComputeName       : 'pkrvmxemqrr7bi4'
==> azure-arm.win-11-ent:  -> Managed OS Disk   : '/subscriptions/xxx/resourceGroups/pkr-Resource-Group-xemqrr7bi4/providers/Microsoft.Compute/disks/pkrosxemqrr7bi4'
==> azure-arm.win-11-ent: Querying the machine's additional disks properties ...
==> azure-arm.win-11-ent:  -> ResourceGroupName : 'pkr-Resource-Group-xemqrr7bi4'
==> azure-arm.win-11-ent:  -> ComputeName       : 'pkrvmxemqrr7bi4'
==> azure-arm.win-11-ent: Powering off machine ...
==> azure-arm.win-11-ent:  -> ResourceGroupName : 'pkr-Resource-Group-xemqrr7bi4'
==> azure-arm.win-11-ent:  -> ComputeName       : 'pkrvmxemqrr7bi4'
==> azure-arm.win-11-ent:  -> Compute ResourceGroupName : 'pkr-Resource-Group-xemqrr7bi4'
==> azure-arm.win-11-ent:  -> Compute Name              : 'pkrvmxemqrr7bi4'
==> azure-arm.win-11-ent:  -> Compute Location          : 'East US'
==> azure-arm.win-11-ent: Generalizing machine ...
==> azure-arm.win-11-ent: Capturing image ...
==> azure-arm.win-11-ent:  -> Image ResourceGroupName   : 'rg-p1-corp-packer-eus'
==> azure-arm.win-11-ent:  -> Image Name                : 'windows11-ent-packer-image-v1-eus'
==> azure-arm.win-11-ent:  -> Image Location            : 'East US'
==> azure-arm.win-11-ent: 
==> azure-arm.win-11-ent: Deleting Virtual Machine deployment and its attached resources...
==> azure-arm.win-11-ent: Deleted -> pkrvmxemqrr7bi4 : 'Microsoft.Compute/virtualMachines'
==> azure-arm.win-11-ent: Deleted -> pkrnixemqrr7bi4 : 'Microsoft.Network/networkInterfaces'
==> azure-arm.win-11-ent: Deleted -> pkrvnxemqrr7bi4 : 'Microsoft.Network/virtualNetworks'
==> azure-arm.win-11-ent: Deleted -> pkrsgxemqrr7bi4 : 'Microsoft.Network/networkSecurityGroups'
==> azure-arm.win-11-ent: Deleted -> pkripxemqrr7bi4 : 'Microsoft.Network/publicIPAddresses'
==> azure-arm.win-11-ent: Deleted -> Microsoft.Compute/disks : '/subscriptions/xxx/resourceGroups/pkr-Resource-Group-xemqrr7bi4/providers/Microsoft.Compute/disks/pkrosxemqrr7bi4'
==> azure-arm.win-11-ent: Removing the created Deployment object: 'pkrdpxemqrr7bi4'
==> azure-arm.win-11-ent: 
==> azure-arm.win-11-ent: Deleting KeyVault created during build
==> azure-arm.win-11-ent: Deleted -> pkrkvxemqrr7bi4 : 'Microsoft.KeyVault/vaults'
==> azure-arm.win-11-ent: Removing the created Deployment object: 'kvpkrdpxemqrr7bi4'
==> azure-arm.win-11-ent: 
==> azure-arm.win-11-ent: Cleanup requested, deleting resource group ...
==> azure-arm.win-11-ent: Resource group has been deleted.
Build 'azure-arm.win-11-ent' finished after 12 minutes 56 seconds.

==> Wait completed after 12 minutes 56 seconds

==> Builds finished. The artifacts of successful builds are:
--> azure-arm.win-11-ent: Azure.ResourceManagement.VMImage:

OSType: Windows
ManagedImageResourceGroupName: rg-p1-corp-packer-eus
ManagedImageName: windows11-ent-packer-image-v1-eus
ManagedImageId: /subscriptions/xxx/resourceGroups/rg-p1-corp-packer-eus/providers/Microsoft.Compute/images/windows11-ent-packer-image-v1-eus
ManagedImageLocation: East US

Reviewing the log output, you can see how Packer is running the deprovisioning inside the virtual machine, capturing the image, and cleaning up all the staging resources.

Within the image resource group, you should now see a new Azure Managed Image.

Testing

After the image is built successfully, it's time to test everything.

1. Navigate to your image and select "Create VM"

2. Navigate through the deployment wizard and set up the virtual machine according to your preferences.

3. Connect into the virtual machine

Now, if we look at the control panel, voila! All apps have been installed. Imagine that you can use this not only for installing apps but also to fully customize your image as needed.

Conclusion

I hope you were able to build a running image by following the steps described in this blog. As I mentioned a few times, what we did here was a very simple setup of Packer. From my experience, a setup like this wouldn't be possible in an enterprise environment with a ton of restrictions. Deploying a public IP alone would not have been possible. However, my plan was to start as simple as possible. We will develop this further and cover some more advanced topics in the future.

Hopefully you get an idea of what is possible with managing images this way.

I leave it up to you to decide if it makes sense to install apps during a Packer build or if you prefer installing them later through a solution like Intune or Endpoint Configuration Manager. In general, I like the image-driven style of deployments, but it really depends on the project and customer needs and preferences. In some cases, it makes perfect sense to bake customizations and applications like monitoring and security agents directly in the image.

Also, consider that this is not limited to client operating systems. In larger environments, it is common to build and provide hardened server images (Linux and Windows) that include all relevant security settings and agents right out of the box.

There are many more exciting topics when it comes to managing images, such as versioning, compute gallery, and automation with pipelines. So, stay tuned!

Inspiration 🎸

As always, here is some inspiration for you. Most of this was written while listening to Meteora (Linkin Park) and The Works (Queen).

Have a great day. 🚀