Why Swarm Intelligence and Genetic Algorithms have a long way to go in Machine learning? | My Neural Path #2

Hey there, curious minds!

This blog is a continuation of the previous one, where I initially optimized Random Forest and SVM using Nature-Inspired Optimization Techniques.

Welcome to the series, “My Neural Path”, where I share my AI experimentation journey- Experiments, Hypothesis, what worked, what didn’t, and what’s coming next.

Since the last blog, I explored several optimization techniques, and Particle Swarm Optimization (PSO) and Mother Tree Optimization (MTO) stood out as the most promising. This time, I evaluated the Baseline model, Optuna-based model, Bald Eagle Optimization (BEO), Particle Swarm Optimization (PSO), and Mother Tree Optimization (MTO) for Random Forest and SVM across six datasets.

1. MNIST Dataset (Handwritten Digits based image dataset)

2. Chinese Dataset (Handwritten Kanji based image dataset)

3. Oxford’s Flower 102 Dataset (Image based 102 different class of flowers)

4. Caltech 101 Dataset (Image based 101 different object classes)

5. Adult’s Census Income (Tabular Dataset)

6. Bank Marketing Dataset (Tabular Dataset)

7. Churn Modelling Dataset (Tabular Dataset)

Among these datasets, Oxford’s Flower 102 and Caltech 101 performed poorly, even after applying PCA to reduce their high dimensionality. This could be due to Random Forest and SVM not being well-suited for image-based datasets. As a result, I dropped them and focused my evaluations on the remaining datasets.

What is Swarm Intelligence?

Swarm Intelligence (SI) is an inspiration from the collective behaviour of natural systems that work in a group or swarm. It focuses on self-organized systems where agents interact with each other and their environment for a common objective. For example -

a) Ant Swarm: Below is a picture where ants are working in a group/swarm to fulfil a common objective which is to reach point B from point A by using themselves as a bridge.

b) Bee Swarm: You will be familiar with the below picture for sure, it’s a bee hive. Bee’s too work in swarms to fulfil a common objective that is to collect and bring pulp from flowers to produce Honey (Worker Bee’s) or protect the Queen Bee’s Nest/Colony (Guard Bee’s).

When the Swarm Intelligence Inspired optimization techniques are made then it is said that these optimization techniques don’t get stuck in local minima and adapt well to a dynamic environment. Some popular SI-based techniques are Particle Swarm Optimization (PSO), Mother Tree Optimization (MTO), Ant Colony Optimization (ACO) etc.

Hypothesis: Swarm Intelligence can outperform standard optimization techniques

It heavily depends on the problem type and optimization criteria whether SI will outperform standard optimization techniques or not. So, I gave it a try to see how it works in our research problem.

The Experiment

1. Dataset:

   a. MNIST (Image based)

   b. Chinese MNIST (Image based)

   c. Adult Census Income (Tabular based)

   d. Bank Marketing (Tabular based)

   e. Churn Modelling (Tabular based)

2. Models used:

   • Random Forest

   • Support Vector Machine (SVM)

Optimization Techniques:

• Optuna (For comparison)

• Bald Eagle Optimization (BEO)

• Particle Swarm Optimization (PSO)

• Mother Tree Optimization (MTO)

Code

# Baseline model
n_estimators: 100

# Optuna parameters used in-general
n_estimators: (50, 300)
max_depth: (5, 50)
max_trial: 30

# Bald Eagle parameters used in-general
population_size = 10
generations = 30
explore_ratio = 0.5 
exploit_ratio = 0.5

# Particle Swarm parameters used in-general
num_particles = 10
num_iterations = 30
w = 0.5  
c1 = 1.5  
c2 = 1.5

# Mother Tree parameters used in-general
num_trees = 10
num_iterations = 30  
alpha = 0.5  
beta = 0.3

# Optuna perimeters used for measuring max capability
n_estimators: (50, 300)
max_depth: (5, 50)
max_trial: 50
# Along with PCA and subsampling at 5000 samples

# Bald Eagle parameters used for measuring max capability
population_size = 20
generations = 50
explore_ratio = 0.5 
exploit_ratio = 0.5 
# Along with PCA and subsampling at 5000 samples

# Particle Swarm parameters used for measuring max capability
num_particles = 20
num_iterations = 50
w = 0.5  
c1 = 1.5  
c2 = 1.5
# Along with PCA and subsampling at 5000 samples

# Mother Tree parameters used for measuring max capability
num_trees = 20
num_iterations = 50  
alpha = 0.5  
beta = 0.3
# Along with PCA and subsampling at 5000 samples

For complete code and experiment setup refer GitHub repo of My Neural Path.

The Results

Experiment on MNIST Dataset

Experiment on Chinese MNIST Dataset

Experiment on Adult Census Income Dataset

Experiment on Bank Marketing Dataset

Churn Modelling Dataset

Nature-Inspired Optimization Techniques did not perform as expected. The main challenge was their high computational cost, making experimentation difficult. While these techniques produced good accuracy, the computational overhead made them inefficient and impractical for optimizing ML models like SVM and Random Forest. Swarm Intelligence-based optimization techniques outperformed evolutionary algorithms like GWO and BEO but failed to surpass standard optimization methods such as GridSearch or Optuna. In some cases, PSO performed slightly better than Optuna, but the difference was marginal and not justifiable given the computational cost.

What’s Next?

For now, I’m wrapping up my experiments on this research problem. In a few days, I’ll make the work available on arXiv, including statistical validation, code, and visualizations. I might revisit Evolutionary Algorithms another time for further experiments. Right now, I’ve figured out an exciting research problem focused on optimizing MoE-based LLM models, and I’m eager to dive into it. Stay tuned for more updates!

Thanks for reading! If you enjoyed this, follow along in the My Neural Path series for more exciting AI experiments!