Docker

Docker Swarm with Elasticsearch Cluster

TL;DR

We use a Docker cluster to launch an Elasticsearch instance as a global service. This will scale our Elasticsearch instance with each machine we add to the cluster. We will set a certain parameters in the Elasticsearch container to launch an Elasticsearch cluster within our Docker cluster.

Context

We created a Docker cluster earlier. We even created an Elasticsearch instance on that cluster. We will be removing that deployment and creating a new instance with clustering functionality.

Removing the current instance

To remove the existing Elasticsearch instance we will be using the following command:

docker service rm elasticsearch

Creating a clustered Elasticsearch instance

To create a new Elasticsearch instance with clustering we will be using the following command:

docker service create \
    --mode global \
    --name elasticsearch \
    --network log_net \
    --mount "type=volume,source=elasticsearch_index,destination=/usr/share/elasticsearch/data" \
    --mount "type=volume,source=elasticsearch_config,destination=/usr/share/elasticsearch/config" \
    --env "cluster.name=docker-cluster" \
    --env "bootstrap.memory_lock=true" \
    --env "ES_JAVA_OPTS=-Xms1g -Xmx1g" \
    --endpoint-mode "dnsrr" \
    --restart-condition any \
    elasticsearch:5-alpine \
    -E "transport.host=0.0.0.0" \
    -E "discovery.zen.ping.unicast.hosts=elasticsearch" \
    -E "discovery.zen.minimum_master_nodes=1"

Here we are adding a few environment variables to tell Elasticsearch that we want to create a cluster. The --mode global option is used to make the instace scalable with the cluster. Another important parameter is endpoint-mode "dnsrr" witch change the container networking mode to DNS round robin. The container parameters after the image configures Elasticsearch to work in cluster mode.

Verification

We can confirm our Elasticsearch cluster by examining the logs of the container. By following these commands we get the logs:

docker service ls # to list the services

docker ps # to list the container on the host

docker container logs <container name> | grep master # to show the logs

#output of the above command
[2017-05-01T14:56:08,806][INFO ][o.e.c.s.ClusterService   ] [T9mVqck] new_master {T9mVqck}{T9mVqcktQ0mb_z7VO18X_w}{u1y7v6lTQYmHCkMHDiQhTQ}{10.0.1.22}{10.0.1.22:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)

We can see the new_master message and nodes joined message that confirms Elasticsearch is working in cluster mode.

Advertisements

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s