Container Orchestrators and Parallel Computing

"Technology is the answer, but what was the question?"

Find it interesting that this quote of Cedric Price is relatable to many situations in technological problem-solving.

Years ago was part of solutioning on Grid computing using OpenMosix and SunGrid clusters. This is not exactly an HPC - but concepts around distributing the jobs, and queuing are similar. HPC needs jobs to be written in a specific way - more on that in another time.

At a high level; HPC is a cluster responsible for scheduling, dispatching, and managing the remote and distributed execution of number of parallel computing "jobs".
HPC is heavily used for long-running engineering simulation jobs wherein there is stuff like message passing, state synchronizing etc going on between different compute nodes.

When I started Kubernetes journey, given the similar sounding component names (in the clustering concepts of HPC and k8s) at the outset - like control plane, workload distribution etc compelled my hippocampus to an analogy with HPC. However, a Container Orchestrator is very much different. It is made for solving a different set of problems.

An orchestrator's purpose is to run containerized "applications"(not "jobs") on a different set of nodes for better resource utilization - but that's not the main goal of a container orchestrator. Container orchestration system solves problems related to automated software deployment, scaling, and management.

This is vastly different from the need of engineers/scientists who submit high throughput computing jobs to a HPC cluster(mostly via a command line) and wait for results after the computing load, distributed to multiple nodes, does its job.

Saw a few articles on combining workloads of HPC on K8s - but it beats me why would one try to do that? The problems each of these technologies solve are different.

As Robert Sapolsky(a neuroendocrinologist) puts it - we do think in "buckets," - term he uses to refer to conceptual filters people employ that keep them locked in particular worldviews.

How do we avoid this trap (an inclination) to try and use a single solution to each and every problem.. even if its not relevant to the problem ..like say happened in the case of blockchain.