Do you really need Kubernetes?

Kubernetes became a kind of standard for container orchestration lately. That's great!. Is extremely flexible, reliable, you can achieve a lot of things and it can be fun :D.

The question is..., does all the companies using it really need it?. I've been thinking about that for a long time, I've worked at a couple of companies that were in the process of implementing it, or already using it. But non of them really needed it. In a nutshell. They were already locked in AWS and they were only running stateless micro-services. Then..., why not to use ECS?.

Recently, I had the chance to interview candidates for DevOps roles..., I kept asking the same question, why your company needs K8s?. Non of them actually gave a clear justification. Guess what..., they were all using stateless micro-services and they didn't have any intention of switching to another cloud provider than AWS (or use a multi-cloud environment). Some of them justified they were using it because that's something widely used and is an important tool for their skill set, cool!.

Does it add value to the company?. Having engineers skilled in a tool they don't really need?. Or probably, thinking out of the box, if the company mentions they are using Kubernetes, will that make that company to be more interesting for new candidates?.

Don't take me wrong, I'm not against using Kubernetes, I just believe it should be a clear answer on the why. You need to be able to justify the extra expenses of running EKS (which have a fixed hourly rate), and justify the extra (human) resources you'll need to maintain it.

Since I was not sure if I was missing something on this reasoning. I've asked a bot friend (ChatGPT) to explain some use cases where you need to use Kubernetes, and you can't use ECS. Bellow the line, is what he (or she) said.

Taking the original question back: Do you really need Kubernetes?.

There are several use cases where Kubernetes might be a better fit than Amazon ECS (Elastic Container Service) due to its flexibility, ecosystem, and features. Here are some scenarios where Kubernetes is preferred or necessary:

1. Multi-Cloud or Hybrid Cloud Deployments

• Use Case: You need to run your applications across multiple cloud providers (e.g., AWS, GCP, Azure) or on-premises data centers.

• Why Kubernetes: Kubernetes offers more flexibility to manage and orchestrate containers across different cloud environments, while ECS is more tightly integrated with AWS. Kubernetes' consistent API and tooling allow for easier deployment across diverse environments.

2. Advanced Networking Requirements

• Use Case: Your application requires custom network plugins, network policies, or service meshes.

• Why Kubernetes: Kubernetes supports advanced networking through its Container Networking Interface (CNI) plugins and can integrate with service meshes like Istio or Linkerd. ECS has more limited networking capabilities compared to Kubernetes.

3. Large Ecosystem and Open-Source Tools

• Use Case: You need to integrate with a rich ecosystem of open-source tools for logging, monitoring, storage, CI/CD, etc.

• Why Kubernetes: Kubernetes has a large, active ecosystem with tools like Prometheus, Grafana, Helm, and many more that offer easy integration. The community and tooling around Kubernetes are broader and more mature than ECS for many scenarios.

4. Custom Scheduler Requirements

• Use Case: Your workload requires a custom scheduling strategy or non-standard resource allocation.

• Why Kubernetes: Kubernetes allows for custom schedulers or configuring scheduling policies with more granularity than ECS. For example, Kubernetes can handle specific node-affinity rules, priority-based scheduling, or complex orchestration patterns that ECS might not support.

5. Greater Control Over Cluster Infrastructure

• Use Case: You want to control the underlying nodes, operating system, and how your containers interact with the infrastructure.

• Why Kubernetes: Kubernetes gives you full control over the worker nodes, operating system versions, and cluster configuration. With ECS (especially with Fargate), this control is more abstracted and limited, so you might not be able to manage certain configurations.

6. Complex Stateful Applications

• Use Case: You need to deploy complex stateful applications like databases or persistent storage-heavy apps.

• Why Kubernetes: Kubernetes has StatefulSets, Persistent Volumes, and other mechanisms that are more robust for managing stateful applications than ECS, which is designed more for stateless microservices.

7. Workloads Requiring Advanced Scaling Strategies

• Use Case: You need more advanced scaling strategies beyond simple service-level scaling, such as horizontal autoscaling based on custom metrics.

• Why Kubernetes: Kubernetes offers powerful horizontal and vertical scaling mechanisms, including scaling based on custom metrics (via the Horizontal Pod Autoscaler) and auto-scaling clusters with different node types (Cluster Autoscaler). ECS’ scaling options are less flexible.

8. CI/CD Integration with Complex Pipelines

• Use Case: You need to integrate with advanced CI/CD pipelines with custom deployment strategies, blue-green deployments, or canary releases.

• Why Kubernetes: Kubernetes supports a wide range of deployment strategies, including blue-green deployments and canary releases, either natively or through third-party tools like ArgoCD and Spinnaker. ECS also supports these to an extent but with less flexibility and fewer native tools.

9. Vendor Lock-In Avoidance

• Use Case: You want to avoid lock-in to a single cloud provider’s ecosystem.

• Why Kubernetes: Kubernetes is an open-source, cloud-agnostic platform that runs on any infrastructure (AWS, GCP, Azure, on-prem). ECS is deeply integrated with AWS, making migration to another cloud or hybrid setup more challenging.

10. Running Edge Computing or IoT Workloads

• Use Case: You need to run workloads on edge devices, in decentralized locations, or across IoT devices.

• Why Kubernetes: Kubernetes, with tools like K3s or KubeEdge, can manage workloads across edge and IoT environments. ECS lacks support for such decentralized, edge-based orchestration out of the box.

In summary, while ECS is great for AWS-centric, simpler use cases with tight integration into AWS services, Kubernetes offers more flexibility, advanced networking, and deployment patterns for multi-cloud, hybrid-cloud, or edge-based deployments.