Generating KUBECONFIG: Automating Role and Cluster Role Creation (RBAC) with Helm and Jenkins on a Self Managed Kubernetes Cluster

In the ever-evolving landscape of cloud-native technologies, Kubernetes has emerged as the de facto standard for container orchestration. Its robustness and flexibility empower organizations to deploy and manage their applications efficiently. However, with great power comes great responsibility, particularly in terms of security. Access control, authentication, and authorization are critical aspects of Kubernetes security that demand meticulous attention.

Accessing a Kubernetes cluster can be accomplished through various methods, including:

1. Kubectl: The primary command-line interface for interacting with Kubernetes clusters. It requires authentication credentials to communicate with the cluster’s API server.

2. API Requests: Applications and services can interact with the Kubernetes API server directly, using client libraries or HTTP requests. Again, authentication is necessary to verify the identity of the requester.

3. Kubernetes Dashboard: A web-based user interface for Kubernetes clusters. It also requires authentication for access.

4. Service Accounts: Kubernetes assigns each pod a service account, enabling applications running within pods to authenticate with the Kubernetes API and perform actions on the cluster.

Authentication and Authorization in Kubernetes:

Authentication is the process of verifying the identity of a user, system, or application attempting to access the Kubernetes cluster. Without proper authentication mechanisms in place, unauthorized entities could gain access to sensitive resources, posing significant security risks.

Authorization, on the other hand, determines what actions a user, system, or application is permitted to perform within the Kubernetes cluster. Even authenticated users must be authorized to perform specific operations, ensuring that they only have access to the resources they need.

In Kubernetes, authentication and authorization are typically handled through various mechanisms:

1. Certificates: Users can authenticate with Kubernetes using client certificates issued by a Certificate Authority (CA). Kubernetes API servers verify these certificates to authenticate users.

2. Tokens: Users can also authenticate using bearer tokens, which are issued by the Kubernetes API server or an external authentication provider. These tokens grant access to specific resources based on configured policies.

3. OIDC (OpenID Connect): Kubernetes supports OIDC authentication, allowing users to authenticate using identity providers such as Google, Microsoft Azure, or Okta. OIDC tokens are exchanged for Kubernetes tokens, which are then used for authorization.

4. Static Password Files: While not recommended for production environments, Kubernetes can authenticate users using static password files. However, this method lacks the scalability and security features of other authentication mechanisms.

5. Static Token Files: Kubernetes supports authentication using static token files. Users can authenticate by presenting a token stored in a file. While this method is straightforward, it lacks the security features and scalability of other authentication mechanisms.

AUTHENTICATION in kubernetes is explained in detail in this official documentation : https://kubernetes.io/docs/reference/access-authn-authz/authentication/

Role-Based Access Control (RBAC) is the primary authorization mechanism in Kubernetes, allowing administrators to define roles and role bindings to control access to cluster resources. Roles specify a set of permissions, while role bindings associate roles with users, groups, or service accounts.

Before RBAC became the standard for authorization in Kubernetes, Attribute-Based Access Control (ABAC) was a commonly used approach. ABAC grants access based on attributes associated with users, such as roles, departments, or locations.

However, ABAC has several disadvantages:

1. Complexity: Managing a large number of attributes and policies can quickly become complex and difficult to maintain.

2. Scalability: As the number of users and resources grows, ABAC policies can become unwieldy and resource-intensive to evaluate.

3. Flexibility: ABAC policies often lack granularity, making it challenging to enforce fine-grained access control.

RBAC, on the other hand, offers several advantages over ABAC:

1. Simplicity: RBAC simplifies access control by defining roles with specific permissions and associating them with users or groups.

2. Scalability: RBAC scales more efficiently, as role definitions are easier to manage and evaluate compared to complex attribute-based policies.

3. Granularity: RBAC allows for finer-grained access control, enabling administrators to specify exactly what actions users can perform on specific resources.

Below is the link of the repository where the files are present:

GitHub - anishsedhaii/rbac-helm: Jenkinfile and Bash script to create role/cluster using Helm Chart…

Let’s start with the Jenkinsfile first.

pipeline {
    agent {
        label "some_agent"
    }
    environment {
      CLUSTER1_MASTER_IP = ""
      CLUSTER2_MASTER_IP = ""
      CLUSTER3_MASTER_IP = ""
      SSH_KEY = "SSH_KEY_PATH"
    }
    parameters {
      choice(name: 'CLUSTER', choices: ['Cluster1', 'Cluster2' , 'Cluster3'], description: 'Choose cluster to implement RBAC')  
      choice(name: 'ROLE', choices: ['ClusterRole', 'Role'], description: 'Choose Role to deploy')
      string(name: 'NAMESPACE', defaultValue: '', description: 'Enter namespace if you want to create role for specific namespace')
      string(name: 'USERNAME', defaultValue: '', description: 'Enter username ex. firstname.lastname')
      string(name: 'EMAIL', defaultValue: '', description: 'Enter email of the user with @organization.com')
      choice(name: 'ACTION', choices: ['plan', 'apply'], description: 'Choose Action')  
    }
    stages {
      stage('Decide Master Node') {
        steps {
          script {
            def MASTER_IP
                    if (params.CLUSTER == 'Cluster1') {
                        MASTER_IP = CLUSTER1_MASTER_IP
                    } else if (params.CLUSTER == 'Cluster2') {
                        MASTER_IP = CLUSTER2_MASTER_IP
                    } else if (params.CLUSTER == 'Cluster3') {
                        MASTER_IP = CLUSTER3_MASTER_IP
                    }
                    env.MASTER_IP = MASTER_IP
                    sh "echo MASTER_IP: ${MASTER_IP}"
          }
        }
      }
      stage('Copy KUBECONFIG from MasterNode') {
        steps {
          dir("${WORKSPACE}") {
          sh """
            scp -i ${SSH_KEY} root@${env.MASTER_IP}:/etc/rancher/rke2/rke2.yaml kubeconfig.yaml
            sed -i "s|https://127.0.0.1:6443|https://${env.MASTER_IP}:6443|g" kubeconfig.yaml
          """
          }
        }
      }
      stage('Deploy RBAC') {
        steps {
          dir("${WORKSPACE}") {
          sh """
            chmod a+x config.sh
            /bin/bash config.sh 
          """
          }
        }
      }
      stage('Extract Config'){
        steps{
          dir("${WORKSPACE}") {
            sh """
              chmod a+x generateconfig.sh
              /bin/bash generateconfig.sh
            """
          }
        }
      }
    }  
    post {
        always {
            script {
                emailext attachmentsPattern: "kubeconfig_user.yaml",
                    body: 'Download the kubeconfig file to connect to the cluster',
                    subject: 'Kubeconfig for Cluster connection',
                    to: "$EMAIL"
            }
        }
      aborted {
          echo 'Pipeline was aborted'
      }
      failure {
          mail to: "aniecesedhai@gmail.com",
          subject: "Failed Pipeline: ${currentBuild.fullDisplayName}",
          body: "Something is wrong with ${env.BUILD_URL}"
      }
    }
}

Stages:

• Decide Master Node: Determines the master node’s IP address based on the selected cluster.

• Copy KUBECONFIG from MasterNode: Copies the kubeconfig file from the master node to the workspace directory and updates the server URL.

• Deploy RBAC: Executes a configuration script (config.sh) to deploy RBAC based on the selected parameters.

• Extract Config: Runs a script (generateconfig.sh) to generate a configuration file.

Post-build Actions:

• Always: Sends an email with the kubeconfig file (kubeconfig_user.yaml) attached, allowing users to download and connect to the cluster.

• Aborted: Prints a message if the pipeline is aborted.

• Failure: Sends an email notification to the specified email address if the pipeline fails, providing details about the failed build.

This Jenkinsfile orchestrates the automation process for deploying Helm charts and configuring RBAC in Kubernetes clusters. However, it’s worth noting that the Jenkins pipeline itself can be skipped if desired, and the provided scripts (config.sh and generateconfig.sh) can be used independently for deploying the Helm chart and generating kubeconfig files.

Now lets move towards the config.sh script

#!/bin/bash

set -xe

ACTION=$ACTION
ROLE=$ROLE
USERNAME=$USERNAME
NAMESPACE=$NAMESPACE
export KUBECONFIG=kubeconfig.yaml
plan () {
    if [ $ROLE == "ClusterRole" ]; then
        helm upgrade $USERNAME --install roles/ --dry-run \
        --set clusterrole.name=clusterrole --set clusterrolebinding.name=clusterrole-binding --set clusterrole.serviceaccount.name=$USERNAME --set clusterrole.secret.name=$USERNAME
    else
        helm upgrade $USERNAME --install roles/ --dry-run \
        --set role.namespace=$NAMESPACE --set role.name=developer-role --set rolebinding.name=developer-role-binding--set role.serviceaccount.name=$USERNAME --set role.secret.name=$USERNAME
    fi
}


apply() {
    if [ $ROLE == "ClusterRole" ]; then
        helm upgrade $USERNAME --install roles/ \
        --set clusterrole.name=clusterrole --set clusterrolebinding.name=clusterrole-binding --set clusterrole.serviceaccount.name=$USERNAME --set clusterrole.secret.name=$USERNAME
    else
        helm upgrade $USERNAME --install roles/ \
        --set role.namespace=$NAMESPACE --set role.name=developer-role --set rolebinding.name=developer-role-binding --set role.serviceaccount.name=$USERNAME --set role.secret.name=$USERNAME
    fi
}
if [ $ACTION == "plan" ]; then
  plan
elif [ $ACTION == "apply" ]; then
  apply
else
  echo "Wrong Option !!"
fi

This bash script serves to deploy RBAC configurations to Kubernetes clusters based on provided parameters. It starts by setting the necessary variables and then defines two functions, plan and apply, to handle the Helm chart deployment in dry-run mode and actual application mode, respectively. The Helm chart creates the following Kubernetes resources:

• Role/ClusterRole

• RoleBinding/ClusterRoleBinding

• ServiceAccount

• Secret

The Secret is bound to the ServiceAccount to store the ServiceAccount token, which serves as a bearer token for accessing the cluster. So which means we are accessing the cluster using Bearer Token method. Here, we have created the secret but it can also be kept in a file and mounted from a volume mount and the path can be provided as client’s key. Also, to make sure that service account tokens do not expire from Kubernetes V1.24 secrets are bound with service account and do not have a expiry(i.e not time bound). However we have not used the TokenSecretAPI to create a token. After the helm chart is deployed we move to the kubeconfig generation part which is then sent via email to the user. I wanted to explain about the apiGroups, Resources and Verbs that have been used on the helm charts but explain it would make this article very long and also kubernetes documentation has explained it properly.

Now comes the kubeconfig.yaml generation part

#!/bin/bash

set -xe

ROLE=$ROLE
MASTER_IP=$MASTER_IP
USERNAME=$USERNAME
NAMESPACE=$NAMESPACE
export KUBECONFIG=kubeconfig.yaml

token_clusterrole(){
  USER_TOKEN=$(kubectl describe secrets "$USERNAME" -n kube-system | awk '/token:/ {print $2}')
}
token_role() {
  USER_TOKEN=$(kubectl describe secrets "$USERNAME" -n "$NAMESPACE" | awk '/token:/ {print $2}')
}
generate_kubeconfig() {
CERTIFICATE=$(kubectl config view --flatten --minify=true -o=jsonpath='{.clusters[].cluster.certificate-authority-data}')
    cat <<EOF > kubeconfig_user.yaml
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: $CERTIFICATE
    server: https://$MASTER_IP:6443 #clusterendpoint
  name: $MASTER_IP #clustername
contexts:
- context:
    cluster: $MASTER_IP #clustername
    user: $USERNAME #service-account
  name: default #clustername
current-context: default
kind: Config
users:
- name: $USERNAME #service-account
  user:
    token: $USER_TOKEN
EOF

    echo "kubeconfig file generated successfully!"
}

if [ $ROLE == "ClusterRole" ]; then
  token_clusterrole
else
  token_role
fi
generate_kubeconfig

This bash script is responsible for generating a kubeconfig file that facilitates user authentication with the Kubernetes cluster. It begins by setting variables and exporting the path to the kubeconfig file.

The generate_kubeconfig function constructs the kubeconfig file using the retrieved user token and the cluster's certificate authority data. This certificate is crucial for establishing a secure connection to the cluster's API server. Additionally, the script sets the context to 'default,' indicating the default configuration for interacting with the cluster.

In summary, this script automates the generation of a kubeconfig file, ensuring secure user authentication and access to Kubernetes cluster resources. It incorporates the user’s token and the cluster’s certificate authority data into the configuration, establishing a secure and authenticated connection for cluster interactions. The default context simplifies cluster interaction by providing a predefined configuration for the user.
Like above we can create a kubeconfig file with multiple context which will allow us to switch between the context using below given command but this is only an example for a single context.

kubectl config use-context context_name

Now the kubeconfig file has been created. The admin who created the kubeconfig file can test the authentication and authorization by using commands given as examples below:

kubectl auth can-i list pods --namespace namespace --as system:serviceaccount:namespace:serviceaccountname
kubectl auth can-i create pods --as username

Please leave some comments if this article has helped you and highlight if there is something that needs to be improved.