Reserving Node Pool Capacity for OKE

If you make heavy use of Auto Scaling for your Oracle Kubernetes Engine (OKE) you may have concerns about your node pools being able to scale up and down without having to worry about running out of capacity. For most customers who are scaling ones and tens of nodes, this may never be a concern, however if you are scaling hundreds or thousands of nodes through the day you may want to be able to take measures to ensure that the capacity you need is ready to go when you need it.

Reserving capacity ahead of time can be done in OCI with capacity reservations in your account. Unlike most providers, OCI allows you to create, and adjust, capacity reservations as you need them, without committing to a long time hourly spend. You can create a capacity reservation now, reserve your instances, and then in a few hours when the instances are no longer needed, you can release them back to the general pool. You get all the benefits that you would expect from a capacity reservation without any of the lock-in that you would expect.

In the event some of your capacity that has been reserved is not in use, you pay 85% of the cost of the instance that is being reserved. Ultimately you DO want to release the reserved capacity when you are done with it. For example, lets say you know that you are going to need 1,000 nodes through the day, your node pools will be scaling up and down through the day. You can create a capacity reservation at the start of the day, use the capacity through the day, and then release it when you head home. This ensures that you have the capacity available through the day, even through your node pools are scaling up and down. And you can take advantage of the 15% discount when the nodes are not in use.

Architecture

Set up an OKE Cluster in a particular region. The region I am using for this exercise (Ashburn) has three availability domains. If the region that you are using has only one Availability Domain, or if you choose to use only a subset of the availability domains, you can do so. You just need to update the python script later on.

Set up capacity Reservations

In each of the Availability domains that you want to use, set up a compute capacity reservation.

Capacity reservations are tied to a specific Availability Domain in regions that have multiple ADs.

You will need to define the Shape and size of the instances that your cluster will use for host nodes. At this point keep all of the nodes the same across the reservations. Also A capacity reservation must be created with a minimum capacity of 1, however you can reduce this to 0 once the full setup process has been completed..

It is helpful to to give the reservation a name that indicated what AD it is being tied to.

You can see here, I have three capacity reservations, each tied to a specific AD. I have also reset the reservation count to 0 for each of these reservations.

Set up node pools

Once I have my three capacity reservations set up, I can create my OKE Cluster. Next I can set up my node pools.

It’s a good idea here to name your node pools in a way that aligns with your reservations. For example in my case the Node Pool “pool3” is going to match to the reservation “AD3-Reservation“ and reside in AD3.

In the node pool placement, I am going to select the corresponding AD, and then under Advanced Options, I am going to choose the Capacity Type as Capacity Reservation, and select the matching Reservation.

Doing this means that the node pool will pull capacity from the reservation, instead of just going to the general pool of nodes for On Demand capacity. Be aware that this is a double edged sword so to speak. Capacity reservations help you to ensure that you have the capacity you need, when you need it, however if your node pool tries to scale out and the capacity reservation does not have enough spare capacity, you will receive an out of capacity error.

This is where my OKENodePoolPrewarmer helps!

The Pre-Warmer

The Pre-Warmer is a Python script, so you will need to have Python installed. It also requires that you have OCI credentials available on your system with permissions to do things like manage capacity reservations.

RESERVATION1 = "ocid1......."
RESERVATION2 = "ocid1......."
RESERVATION3 = "ocid1......."

First step to use this script is to find the Ocid for your capacity reservations. Each of your capacity reservations will have it’s own OCID. You can obtain this by looking at the details for the capacity reservation. If you are in a region that has only one Availability Domain, or you have chosen to use less availability domains then are available, you may have only 1 or 2 entries in this list.

The next thing to do is configure the operation of the script.

reservationCount = 3    # Number of reservations to monitor
desiredCapacity = 100   # Desired total capacity to reach
minimumCapacity = 90    # Minimum capacity acceptable
checkInterval = 1       # Interval in seconds to check the capacity
maximumTests = 10       # Maximum number of times we will check each reservation before giving up.

reservationCount is the number of capacity reservations that you are using for your nodes. In my case above, I am using 3 capacity reservations. You should set this number to the number of reservation OCIDs that you specified in the previous step.

desiredCapacity is the number of host nodes that you would like to spin up. This is your ideal situation.

minimumCapacity is the minimum number of nodes that you will accept. I have added this as an option because in some cases, you may want to have a certain number of nodes, but you may be able to operate with a lower number of nodes in a pinch.

checkInterval is the number of seconds that we will wait between checking the reservations to see if they have filled to the desired capacity. In this case above, we will check every 1 second.

maximumTests is the maximum number of times we will check each reservation before we give up and move on to the next reservation.

Operation

Once the script is fired off, we are going to take the total number of requested nodes, and divide that evenly across the number of reservations. So for example if you want 100 nodes across 3 ADs you will get 34 nodes in each reservation. 100 / 3 = 33.33333 which then gets rounded up to 34.

The script will then check each reservation, and add the required number of nodes to the reservation. At the point of writing this, the script does not take into account existing unused capacity in the reservation. So if your current reservation has a capacity of 10, with 5 nodes unused, in the above situation the capacity reservation will be updated to 44 nodes, not 39.

Once all the node pools have been updated, the script will loop around checking the reservations to see when they reach the desired capacity, or they time out based on the maximum number of attempts. Note that for large numbers of nodes you may want to increase the check interval and/or maximum tests.

Output

Once the script completes, you will get the following JASON output

{
    'capacityMode': 'Desired', 
    'reservationCount': 3, 
    'desiredCapacity': 100, 
    'minimumCapacity': 31, 
    'perADCapacity': 35, 
    'safeCapacity': 105
}

Capacity mode will be one of the following options:

• Desired - At least the desired number of nodes were reserved

• Minimum - At least the minimum number of nodes were reserved, but less than the desired number of nodes

• Failed - The minimum number of nodes were not reserved during the checking period.

safeCapacity will indicate the maximum node size that is safe to launch.

safeCapacity is based on the reservation with the least number of successful nodes times the number of reservations. This could mean that the safeCapacity is less than the number of nodes reserved. This is because the Node Auto Scaler will attempt to balance the requested nodes across the various node pools for resiliency. We do not want a situation where one reservation runs out of capacity before the other reservations, and causes an out of capacity error.

Considerations

This script sets the desired capacity across your reservations. It is possible that the node pools do not fill in the time that the script is checking the capacity, but that additional nodes becomes available after and are reserved. This will mean that your reservation will have additional nodes added after the fact.

Due to rounding up of the number of nodes, you may get more nodes reserved than you initially requested.

Once your work has been completed and you no longer require the worker nodes, the node pool will return the capacity to the reservation. These nodes will continue to be reserved, and you will continue to pay for the nodes (85%) as long as they are reserved. It is important to release the reservations when no longer required.