Elastic search multi node cluster with docker-compose

A multi-node Elasticsearch cluster is a group of Elasticsearch nodes that work together to distribute data and provide high availability and scalability for search operations. This type of cluster is useful for large-scale applications that need to handle a lot of data and search requests.

Types of clusters

There are two main types of multi-node Elasticsearch clusters

• Master-Only: Each node acts as both a master and a data node.

• Master-Masons: Each node is either a master node or a data node.
  Master nodes manage the cluster and distribute data, while data nodes hold the actual data and serve search requests.

Setting up a multi-node cluster 🧩

To set up a multi-node cluster we will deploy 3 elastic search services with help of docker-compose. One of the services will be our main service and the other two will be helper services.

An alternative approach to the master node will let elastic search decide the master node by providing multiple node addresses to select from. For simplicity of this topic, let's make the first ES service the master node.

version: '3.1'
services:
  es_service_1:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10
    container_name: es_container_1
    environment:
      - cluster.name=es_cluster
      - node.max_local_storage_nodes=3
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.security.enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./es_data_1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic_network

  es_service_2:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10
    container_name: es_container_2
    environment:
      - cluster.name=es_cluster
      - node.max_local_storage_nodes=3
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.security.enabled=false
      - discovery.zen.ping.unicast.hosts=es_container_1
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./es_data_2:/usr/share/elasticsearch/data
    networks:
      - elastic_network

  es_service_3:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10
    container_name: es_container_3
    environment:
      - cluster.name=es_cluster
      - node.max_local_storage_nodes=3
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.security.enabled=false
      - discovery.zen.ping.unicast.hosts=es_container_1
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./es_data_3:/usr/share/elasticsearch/data
    networks:
      - elastic_network

networks:
  elastic_network:
    driver: bridge

The above example has one main elastic service es_service_1 which listens on port 9200 and the other two services es_service_2 and es_service_3 communicate with es_service_1 via docker network.

discovery.zen.ping.unicast.hosts provides a list of the addresses of the master-eligible nodes in the cluster. In the above example only one address es_container_1 however multiple addresses can be provided (comma-separated strings)

Elasticsearch cat APIs are designed for people who need compact and aligned text at a terminal. The cat nodes API returns information about a cluster’s nodes :

curl -X GET localhost:9200/_cat/nodes

---

Wrap up! 🎁

This is one of the topics I face in my everyday life as a Software Engineer.
Stay tuned for more such content.
If you find it helpful, share this article so others can find it too.

Adios. 👋

Connect with me:
Linkedin