Leveraging Amazon FSx as Persistent Volume for EKS Windows Nodes

Introduction to Amazon FSx

In the ever-evolving landscape of cloud computing, managing and scaling storage solutions for containerized applications is a critical aspect of modern infrastructure. One of the challenges, particularly in the context of Windows containers orchestrated by Kubernetes, is the efficient provisioning and management of persistent storage. In this blog post, we delve into the solution provided by Amazon FSx for Windows File Server.

What is Amazon FSx?

Amazon FSx is a fully managed file storage service that makes it easy to set up and scale file systems in the AWS Cloud. Specifically designed to address Windows workload requirements, Amazon FSx for Windows File Server provides a shared file storage solution that can be seamlessly integrated with Windows applications and services.

Prerequisites

1. Amazon EKS cluster with windows nodes configured.

2. Amazon FSx for Windows File Server deployed (Terraform module for FSx). Make sure to enable inbound port 445 in the Security Group for FSx.

3. Microsoft Active Directory domain deployed to support Amazon FSx for Windows File Server.

4. Roles and policies attached to EKS cluster for accessing Amazon FSx.

Using the SMB CSI Driver on Amazon EKS Windows nodes

CSI drivers such as SMB CSI Driver allows Amazon Elastic Kubernetes Service (Amazon EKS) clusters to manage Amazon FSx shares lifecycle for persistent volumes. The Server Message Block (SMB) protocol is a network file sharing protocol that allows applications to access files or other resources at a remote server, here FSx.

1. Installing the SMB CSI Driver

   Click here to install SMB CSI Driver on your EKS Windows nodes. Apply all these manifest files using kubectl apply command.

2. Create a Kubernetes secret

   Windows nodes needs Read/Write permissions in the SMB share in order to offer it as local directories to the Windows pod. Create a secret named "smbcreds" that contains an Active Directory username and password with Read/Write privileges on the Amazon FSx for Windows File Server share.

    kubectl create secret generic smbcreds --from-literal domain=DOMAINNAME --from-literal username=USERNAME --from-literal password=PASSWORD
   

   Replace with the following:
   DOMAINNAME: The Active Directory FQDN domain to which the Amazon FSx for Windows File Server is joined.
   USERNAME: The domain user name with Read/Write access to the Amazon FSx for Windows File Server root share.
   PASSWORD: The password for the specified user.

3. Mounting Amazon FSx file share

   You can check whether the mount of your Amazon FSx file share on your Windows node is successful or not. Connect to your windows instance and run the command Test-NetConnection FSx_ipaddress -Port 445 in PowerShell to check if the connection works. Replace FSx_ipaddress with your File Server IP address.

4. Test by mounting local directories in the Windows Pod using SMB CSI Driver on Deployments with PV and PVC.

   1. Create a PersistentVolume manifest. Save it as pv-smb.yaml.

       apiVersion: v1
       kind: PersistentVolume
       metadata:
         name: fsx-pv
         annotations:
           pv.kubernetes.io/provisioned-by: smb.csi.k8s.io
       spec:
         capacity:
           storage: 30Gi  
         accessModes:
           - ReadWriteMany  
         persistentVolumeReclaimPolicy: Retain
         storageClassName: fsx-sc
         mountOptions:
           - dir_mode=0777
           - file_mode=0777
         csi:
           driver: smb.csi.k8s.io
           readOnly: false
           volumeHandle: fs-302kai605a50028b4    #Replace it with your FSx ID
           volumeAttributes:
             source: //10.4.20.19/share    #Replace IP with your File Server IP Address
           nodeStageSecretRef:
             name: smbcreds
             namespace: default
      

   2. Apply it using kubectl apply -f pv-smb.yaml

   3. Create a PersistentVolumeClaim manifest. Copy the following manifest and save it as pvc-smb.yaml:

       apiVersion: v1
       kind: PersistentVolumeClaim
       metadata:
         name: fsx-pvc
       spec:
         accessModes:
           - ReadWriteMany
         resources:
           requests:
             storage: 25Gi 
         volumeName: fsx-pv
         storageClassName: fsx-sc
      

   4. Apply it using kubectl apply -f pvc-smb.yaml

   5. Deploy a pod that consumes the PersistentVolumeClaim. Copy the following manifest and save it as busybox-smb.yaml

       apiVersion: apps/v1
       kind: Deployment
       metadata:
         name: busybox-smb
         labels:
           app: busybox
       spec:
         replicas: 2
         template:
           metadata:
             name: busybox
             labels:
               app: busybox
           spec:
             nodeSelector:
               "kubernetes.io/os": windows
             containers:
               - name: busybox
                 image: e2eteam/busybox:1.29
                 command:
                   - "powershell.exe"
                   - "-Command"
                   - "while (1) { Add-Content -Encoding Ascii C:\\mnt\\smb\\data.txt  $(Get-Date -Format u); sleep 1 }"
                 volumeMounts:
                   - name: smb
                     mountPath: "/pv/fsx-pv"
             volumes:
               - name: smb
                 persistentVolumeClaim:
                   claimName: fsx-pvc
         selector:
           matchLabels:
             app: busybox
      

   6. Apply it using kubectl apply -f busybox-smb.yaml

   7. To validate if SMB CSI Driver was correctly setup, lets proceed with a simple test of writing a simple “Hello” file to the local directory “C:\pv\fsx-pv” inside Pod1 and accessing the file from Pod2.

      1. Identify busybox pods name -> kubectl get pods

      2. Go inside the Pod1 -> kubectl exec -it busybox-smb-POD-NAME-1 -- powershell

      3. Create a text file -> Write-Output "Hello, world!" | Out-File -FilePath "C:\pv\fsx-pv\hello.txt"

      4. Exit Pod1 and access Pod2 -> kubectl exec -it busybox-smb-POD-NAME-2 -- powershell

      5. You will see hello.txt at “C:\pv\fsx-pv” inside Pod2 -> Get-Content -Path "C:\pv\fsx-pv\hello.txt"

Conclusion

In this post, we have outlined how to configure your Amazon EKS Windows cluster pods to mount a volume through an SMB share hosted in Amazon FSx for Windows File Server. We then used a third-party CSI SMB driver to deploy a sample application to validate the cross-access of the file share. The CSI driver is a free, open-source Kubernetes CSI driver and with no additional cost on the environment.

This approach of data persistence demonstrates the capabilities of extending the use of persistent storage hosted on SMB shares with Amazon FSx using Container Storage Interface (CSI) to access Windows shares storage.