Build a Multi-tenant Kubernetes Cluster

Ganesh BalimidiGanesh Balimidi
14 min read

Table of contents

Scenario Description:

In this Scenario, you will assume the role of a Kubernetes Engineer in the Automation team of a Marketing company. The company relies on Kubernetes to power its operations. Currently, each internal department has its own dedicated cluster, leading to high costs and management overhead. To address this issue, the Chief Technology Officer (CTO) has proposed transitioning to a Multi-tenant architecture, where a single cluster would serve multiple teams within the company. Your task is to implement a proof-of-concept to assess the feasibility and advantages of this project.

As part of the proof-of-concept, you will be responsible for provisioning users and groups within the multi-tenant cluster. Additionally, you will need to configure limits and constraints for these users to ensure a stable and efficient environment across different teams. This will help evaluate the viability of the multi-tenant approach and its benefits for the organization.

Objectives

  • Apply multi-tenancy principles to build a shared Kubernetes cluster

  • Generate, sign, and approve certificates using Kubernetes

  • Create users and groups to provide fine-grained access to shared clusters

  • Define RBAC rules to restrict permissions on users and groups

  • Limit the use of Kubernetes resources through Limit Ranges and Resource Quotas

Requirement

  • Basic knowledge of Kubernetes

  • Basic understanding of Pods and Deployments

What you'll learn

  • Role-Based Access Control

  • Multi-tenancy Principles

  • Limit Ranges and Resource Quotas

  • User and Group Management in Kubernetes

Your project assignment

Your Manager:

Hello Ganesh! Thank you for taking on this project for our team. After facing high cloud bills due to multiple Kubernetes clusters, we have decided to embark on a cost-saving initiative. Our CTO has proposed consolidating all our clusters into a single cluster using Multi-tenancy principles. As Kubernetes Engineer, we rely on your expertise to conduct a proof-of-concept and merge these clusters successfully.

There are four essential principles outlined by the CTO that we need to follow for a seamless transition:

  1. User and Group Creation: Find a way to create users and groups in Kubernetes to generate specific users and groups for each team.

  2. Namespace Isolation: Each team should have its dedicated namespace, and access to other namespaces should be strictly forbidden for team members.

  3. Restricted Resources: Within each namespace, only Deployments, ReplicaSets, and Pods should be allowed as resources.

  4. Resource Limitations: Each namespace must enforce restrictions such as not allowing the creation of more than 5 pods. Additionally, each pod should be limited to a maximum of 20Mi of RAM and 10m of CPU.

Considering these points, please proceed with the implementation by creating a single user named John, who is part of the development team. Best of luck with the project!

Task 1: User Creation in Kubernetes

In this task, your objective is to create a user named John for the development team in Kubernetes. The purpose of this task is to assess the process of user creation, including assigning the user to a specific group, and to test the login functionality for the newly created user.

Please note that Kubernetes does not provide a native API for user creation. Therefore, you will need to rely on an external authentication method, such as self-signed certificates, to create the user.

It is important to keep in mind that in this task, you are only required to create the user and test its login. You may encounter errors during the process, such as "User 'xyz' cannot list resource 'xyz'". This error indicates that the user is valid but does not yet have any permissions assigned to it.

However, it is unacceptable to encounter errors similar to "error: You must be logged in to the server (Unauthorized)", as this implies that the new user is unable to log in to the cluster.

Please proceed with creating the user John for the development team and ensure to verify successful login functionality.

Task 2: Create a Namespace for the Team and Grant RBAC Permissions to Users

In this task, your objective is to create a namespace specifically for the development team, ensuring that all their resources are isolated within this namespace. You also need to grant RBAC (Role-Based Access Control) permissions to users, specifically John, to ensure that he can only access the development namespace. Additionally, John should have the ability to perform all operations on Deployments, ReplicaSets, and Pods.

It is crucial to restrict John from performing any operations on resources other than the mentioned ones. For instance, John should not have the capability to create configmaps or secrets. You can test this by attempting to create a dummy configmap/secret, which should result in an error.

Please proceed with creating the development namespace and configuring RBAC permissions for John accordingly. Ensure that John can only access the development namespace and has the necessary privileges for the specified resources.

Task 3: Limit the Amount of Resources and Pods in the Development Namespace

In this task, your goal is to enforce limitations on the amount of resources and pods that can be consumed by the development team, including John. The specific constraints are as follows: each pod in the development namespace should be limited to a maximum of 32Mi of RAM and 20m of CPU. If a pod attempts to exceed these limits, its creation should fail.

To validate the functionality, please create a dummy deployment using the nginx:alpine image. During the creation of this deployment, ensure that the aforementioned conditions are enforced. This means that the deployment should only succeed if the resource and pod limitations are not violated.

Please proceed with creating the dummy deployment and verifying that the restrictions on resource consumption and pod limits are properly enforced within the development namespace.

Task 4: Limit the Amount of Pods Running in the Development Namespace

In this task, the objective is to enforce a limitation on the maximum number of pods that can be created by the development team in the development namespace. The restriction states that the team should not be able to create more than 5 pods simultaneously. This ensures fair resource allocation and prevents one team from monopolizing all available resources in the cluster.

To test this limitation, you should apply the rule as the admin user. Afterwards, you can log in as "John" and attempt to create more than 5 pods. If the rule is functioning correctly, John will not be able to create a sixth pod while the restriction is in place.

Please proceed with implementing the rule to limit the number of pods in the development namespace. Once applied, log in as "John" and attempt to create additional pods to verify that the restriction is correctly enforced.

Solution:

Task 1: User Creation in Kubernetes

1. Create a directory for the team and the user. This is not a requirement, but it simplifies and logically organizes teams and users.

$ mkdir -p ~/teams/development/john

2. Create a private key for the user John using OpenSSL.

# Changes directory
$ cd ~/teams/development/john

# Creates the private key
$ openssl genrsa -out john.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
...
$ ls -l
john.key # You should see the key here

3. Create a certificate signing request. You do this to let Kubernetes know that you'd like to trust this new john.key certificate, so that it can log in and access the cluster as a valid user.

$ cd ~/teams/development/john
# Create file that holds the certificate signing request
## Note how the group is passed in the form of "/O=GROUP_NAME"
$ openssl req -new -key john.key -out john.csr -subj "/CN=john/O=development"

# Check that the csr file was created
$ ls
john.csr  john.key # See john.csr

4. Create a CertificateSigningRequest object in Kubernetes to request the approval of the new certificate for John.

cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:  
  name: john-development # Any name can be used here. 
spec:  
  signerName: kubernetes.io/kube-apiserver-client   
  groups:  
  - system:authenticated  
### Be sure to pass the correct file name.  
request: $(cat john.csr | base64 | tr -d '\n')  
usages:  
- digital signature  
- key encipherment  
- client auth
EOF

# Check the CertificateSigningRequest objects. You will see more fields
# in the output, but below are the important ones
$ kubectl get CertificateSigningRequest
NAME               REQUESTOR        CONDITION
john-development  system:admin     Pending # This should be "Pending"

5. Now that you as a Kubernetes admin have sent a request to approve the "creation of the user John" - or more specifically, the approval of his certificate - it's time to approve the request.

# Approve the request
$ kubectl certificate approve john-development

# Ensure that the certificate was approved and issued
$ kubectl get CertificateSigningRequest
NAME               REQUESTOR        CONDITION
john-development  system:admin     Approved,Issued  # This should be "Approved, Issued"

# Retrieve the generated certificate for John
$ cd ~/teams/development/john

$ kubectl get csr john-development -o jsonpath='{.status.certificate}'  | base64 -d > john.crt

# The content of the certificate should look like below
$ cat john.crt 
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

6. Congrats! You have successfully granted John access to the Kubernetes cluster. It's time to configure the access for John using the certificates that you approved above.

$ cd ~/teams/development/john
# Configure the certificates for the user john
$ kubectl config set-credentials john \
  --client-certificate=$PWD/john.crt \
  --client-key=$PWD/john.key

# Add John as a new "user" - to be more specific, it'll be added as a kubernetes context to your kubeconfig file.
$ kubectl config set-context john \
  --cluster=default \
  --user=john

# List the available "users" (contexts) in your system and you should now see john's one poping up.
$ kubectl  config get-contexts
CURRENT   NAME      CLUSTER   AUTHINFO   NAMESPACE
*         default   default   default    
          john      default   john

7. Switch to the "john" context and try to access any resource in Kubernetes.

$ kubectl config use-context john 
Switched to context "john".
# Check your current context
$ kubectl  config get-contexts
CURRENT   NAME      CLUSTER   AUTHINFO   NAMESPACE
          default   default   default    
 *        john      default   john    

$ kubectl  get pods 
Error from server (Forbidden): pods is forbidden: User "john" cannot list resource "pods" in API group "" in the namespace "default"

It's important to notice that the error clearly states: User "john" cannot list resource "XYZ", which means that John is a valid user within Kubernetes but needs some additional configuration in terms of permissions. But, the main goal of this task was to create a valid user, and it's now done!

If you receive an error with a different message, please check that you followed all the steps as defined above.

Note: Since John doesn't have any permissions, yet, switch back to the admin context.

$ kubectl config use-context default
$ kubectl  config get-contexts
CURRENT   NAME      CLUSTER   AUTHINFO   NAMESPACE
*         default   default   default    
          john      default   john

Task 2: Create a Namespace for the Team and Grant RBAC Permissions to Users

1. Create the development namespace. Ensure that you run these commands as the admin user.

$ kubectl config use-context default
Switched to context "default". 

# Ensure you're using the default context
$ kubectl  config get-contexts
CURRENT   NAME      CLUSTER   AUTHINFO   NAMESPACE
*         default   default   default    
          john      default   john

# Create the namespace
$ vi ~/teams/development/k8s.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: development

# Save the file and exit

# Apply with kubectl
$ kubectl apply -f ~/teams/development/k8s.yaml
namespace/development created

2. Create a role that describes the permissions that the user John will have. Remember these permissions should be scoped to the development namespace.

$ vi ~/teams/development/k8s.yaml
...
kind: Namespace
...
--- # Add these three hyphens to add a new object definition
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: development        # Scoped namespace
  name: development-team        # An arbitrary name
rules:
- apiGroups: ["", "extensions", "apps"]

  # Allow ONLY these resources
  resources: ["deployments", "replicasets", "pods"]

  # Allow ALL operations on the above resources
  verbs: ["*"]

# Save the file and exit

3. Bind the role that you created above with the development group that you configured when you created the certificates for John. It's important to bind the role to the group "development" so that you can add more users to this group in the future.

$ vi ~/teams/development/k8s.yaml
...
kind: Role
metadata:
  namespace: development        # Scoped namespace
  name: development-team        # An arbitrary name
...
--- # Add a new object
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: development      # Allowed namespace
  name: development-team      # An arbitrary name
subjects:
- kind: Group
  name: development          # development group here
  apiGroup: "rbac.authorization.k8s.io"
roleRef:
  kind: Role
  name: development-team      # role name here
  apiGroup: rbac.authorization.k8s.io

# Save the file and exit

# Apply the config

$ kubectl apply -f ~/teams/development/k8s.yaml
...
role.rbac.authorization.k8s.io/development-team created
rolebinding.rbac.authorization.k8s.io/development-team created

4. Become John and test the permissions. Remember that you granted permissions to Deployments, ReplicaSets and Pods ONLY within the development namespace.

$ kubectl config use-context john
Switched to context "john".

# Ensure that you became John
kubectl  config get-contexts
CURRENT   NAME      CLUSTER   AUTHINFO   NAMESPACE
          default   default   default    
*         john      default   john     

# Test permissions on the development namespace
$ kubectl get deploy,rs,pods -n development
No resources found in development namespace. # Works!

# Test permissions for something other than deploy,rs,pods for John
$ kubectl get cm,secrets -n development

# John shouldn't be able to access configmaps or secrets
Error from server (Forbidden): configmaps is forbidden ...
Error from server (Forbidden): secrets is forbidden ...

# Test the permisions on another namespace
$ kubectl get deploy,rs,pods -n default

# John shouldn't be able to access resources that belong to other namespaces
Error from server (Forbidden): deployments.apps is forbidden ...
Error from server (Forbidden): replicasets.apps is forbidden ...
Error from server (Forbidden): pods is forbidden ...

Task 3: Limit the Amount of Resources and Pods in the Development Namespace

1. Ensure that you become the admin user to be able to create the restrictions in the namespace.

$ kubectl config use-context default
Switched to context "default".

2. Create the constraints in the development namespace with 20m for CPU and 32Mi for RAM.

$ vi ~/teams/development/k8s.yaml
...
kind: RoleBinding
metadata:
  namespace: development
  name: deployment-team
...
---  # Add a new object to limit the resources
apiVersion: v1
kind: LimitRange
metadata:
  name: development-limits # Arbitrary name
  namespace: development   # Namespace to constrain
spec:
  limits:
    - type: "Container"
      max:
        cpu: 20m
        memory: 32Mi

# Save and exit

# Apply using kubectl
$ kubectl apply -f ~/teams/development/k8s.yaml
...
limitrange/development-limits created

3. Create a deployment to test that the development team is constrained in terms of pod resources. This deployment will ask for more CPU and RAM than the values allowed in the constraint, and it should fail.

$ vi ~/teams/development/pods-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-test
  namespace: development # development namespace
spec:
  replicas: 2
  selector:
    matchLabels:
      app: deployment-test
  template:
    metadata:
      labels:
        app: deployment-test
    spec:
      containers:
      - name: nginx
        image: nginx:alpine # Use a tiny image
        resources:
          requests:
            cpu: 100m       # Ask for lots of cpu
            memory: 200Mi   # Ask for lots of RAM

# Save the file and exit

# Apply the file with kubectl as John
$ kubectl config use-context john
Switched to context "john".

$ kubectl apply -f ~/teams/development/pods-demo.yaml
deployment.apps/deployment-test created

# Ensure that the pods are not created, because the violate the contraints
$ kubectl get -f ~/teams/development/pods-demo.yaml -o yaml
status:
  conditions:
  ...
    message: 'Pod "deployment-test-56c87dc4d5-nx58b" is invalid: [spec.containers[0].resources.requests:
      Invalid value: "100m": must be less than or equal to cpu limit, spec.containers[0].resources.requests:
      Invalid value: "200Mi": must be less than or equal to memory limit]'

4. Provide valid values to ensure that members of the development team can create pods within the enforced limits.

$ vi ~/teams/development/pods-demo.yaml
apiVersion: apps/v1
kind: Deployment
...
    spec:
      containers:
      - name: nginx
        image: nginx:alpine # Use a tiny image
        resources:
          requests:
            cpu: 10m       # Ask for cpu within the limits
            memory: 10Mi   # Ask for ram within the limits

# Save the file and exit

# Apply with kubectl
$ kubectl apply -f ~/teams/development/pods-demo.yaml

# Check the pods
$ kubectl  get pods -n development

# Works@
NAME                               READY   STATUS    RESTARTS   AGE
deployment-test-6647479965-7jhd5   1/1     Running   0          19s
deployment-test-6647479965-jgd64   1/1     Running   0          10s

Task 4: Limit the Amount of Pods Running in the Development Namespace

1. Ensure that you run these commands as the admin user

$ kubectl config use-context default
Switched to context "default".

2. Modify the k8s.yaml file and add a resource quota to limit the number of pods in the development namespace.

$ vi ~/teams/development/k8s.yaml
--- # Add a new component at the end of the file
apiVersion: v1
kind: ResourceQuota
metadata:
  name: development-pods-limits
  namespace: development
spec:
  hard:
    pods: "5"
# Save the file and exit

# Apply the config
$ kubectl apply -f ~/teams/development/k8s.yaml
...
resourcequota/development-pods-limits created

3. Become John and test the new constraint by creating 6 pods with the below command:

$ for i in {1..6}; do kubectl -n development run max-pods-$i --image=nginx:alpine ; done

pod/max-pods-1 created
pod/max-pods-2 created
pod/max-pods-3 created
Error from server (Forbidden): pods "max-pods-4" is forbidden: exceeded quota
Error from server (Forbidden): pods "max-pods-5" is forbidden: exceeded quota

# Check how many pods were actually created
$ kubectl  get pods -n development
NAME                               READY   STATUS    RESTARTS      AGE
deployment-test-6647479965-7jhd5   1/1     Running   1 (38m ago)   42h
deployment-test-6647479965-jgd64   1/1     Running   1 (38m ago)   42h
max-pods-1                         1/1     Running   0             98s
max-pods-2                         1/1     Running   0             98s
max-pods-3                         1/1     Running   0             98s

Great job! By successfully configuring multitenancy in the Kubernetes cluster, you have achieved the goal of providing secure and isolated environments for different teams within separate namespaces. The implemented rules and limitations ensure that each team can operate within their designated namespace while adhering to resource constraints.

This accomplishment marks an important milestone in optimizing resource utilization, enhancing security, and facilitating efficient management of the Kubernetes cluster. Well done on successfully completing the task and enabling multitenancy in the cluster!

Check out my previous post about Create a Kubernetes POD based on Custom Specifications

Learn More About Kubernetes and Devops here

0
Subscribe to my newsletter

Read articles from Ganesh Balimidi directly inside your inbox. Subscribe to the newsletter, and don't miss out.

clusterrolebindingsself signed certificatesrolesKubernetes clsuterKubernetes namespaces

Written by

Ganesh Balimidi
Ganesh Balimidi