Adding Redundancy to Consul, Nomad and Gluster

This is Part 5 of the Avoiding the Cloud series. This step is independent of the provider you use, as long as you can reproducibly provision Salt Minions like we did in Part 2.

In Part 1: Building a Docker cluster with Nomad, Consul and SaltStack on TransIP we defined our technology stack, our IP address space and node types and roles.

In Part 2: Reproducibly provisioning Salt Minions on TransIP we provisioned 4 different VPSs with just a Salt Minion and their hostnames and IP address.

In Part 3: Provisioning Consul, Nomad and Gluster with SaltStack you created a central SaltStack configuration that defined and configured all nodes to their correct configuration.

In Part 4: Getting Traefik, Health checks and Dashboards on a Nomad Cluster you made your Traefik, Consul and Nomad dashboards available to yourself and you made sure your provider's load balancer could direct traffic to your cluster.

In this part of the series we will take all the steps to deploy additional nodes to our cluster to create redundancy. So we'll go from 1 to 3 Consul servers, from 1 to 3 Nomad servers, from 1 to 2 Docker hosts and from 1 to 3 Gluster servers.

Just like in the other parts, whenever I talk about putting specific contents in a file, I'll leave it up to you to pick your preferred way. Personally I use vi, but nano is present as well.

Getting new nodes ready

It's time for you to step back to Part 2: Reproducibly provisioning Salt Minions on TransIP again, to provision 7 new nodes:

• 2 Consul nodes with names consul-server-02 and consul-server-03
• 2 Nomad nodes with names nomad-server-02 and nomad-server-03
• 1 Docker node with name docker-02
• 2 Gluster nodes with names gluster-02 and gluster-03

In short you have to take the following steps:

1. Order 7 new nodes (We used BladeVPS /X4 at TransIP before)
2. Add all 7 nodes to your private network, so they can communicate with your Salt master
3. Add their target configuration to config.php
4. Use install_vps.php to create Salt Minions out of the nodes
5. Accept them as Salt Minions with the salt-key step
6. And of course don't forget to do the manual steps for the Gluster nodes at the end to split the disk partition.

From this point on I'll assume there are 7 new Salt minions available for us to work with.

Adding redundancy to Consul

Let's first adapt our cluster configuration in /srv/salt/config.jinja for our larger cluster. That means changing the consul_server_ips parameter to:

    "consul_server_ips": ["192.168.0.10", "192.168.0.11", "192.168.0.12"],

This will make sure our clients will always try to connect with surviving members of the Consul cluster when they need to. In addition we'll update the bootstrap_expect parameter in /srv/salt/data/consul.d/server.hcl to:

bootstrap_expect = 3

Now let's make SaltStack do what it does best: Fix the configuration state of our current and newest consul server minions!

$ sudo salt 'consul-server-*' state.apply

After this step finished we can check our Consul cluster membership:

$ sudo salt 'consul-server-01' cmd.run 'consul members'
consul-server-01:
    Node               Address             Status  Type    Build  Protocol  DC   Segment
    consul-server-01   192.168.0.10:8301   alive   server  1.9.4  2         dc1  <all>
    consul-server-02   192.168.0.11:8301   alive   server  1.9.4  2         dc1  <all>
    consul-server-03   192.168.0.12:8301   alive   server  1.9.4  2         dc1  <all>
    docker-01          192.168.0.30:8301   alive   client  1.9.4  2         dc1  <default>
    nomad-server-01    192.168.0.20:8301   alive   client  1.9.4  2         dc1  <default>
    gluster-01         192.168.0.100:8301  alive   client  1.9.4  2         dc1  <default>

Well done.. Suddenly you have a Consul Cluster with redundancy!

Adding redundancy to Nomad

Adding redundancy to our Nomad server is even easier. Because we installed Consul Service discovery, Nomad can already find it's servers and clients. So we don't even have to adapt any configuration files, other than increasing the expected number of servers by updating the bootstrap_expect parameter in /srv/salt/data/nomad.d/server.hcl to:

  bootstrap_expect = 3

Now let's make SaltStack do what it does best: Fix the configuration state of our current and newest consul server minions!

$ sudo salt 'nomad-server-*' state.apply

After this step finished we can check our Nomad cluster membership:

$ sudo salt 'nomad-server-01' cmd.run "nomad server members"
nomad-server-01:
    Name                    Address       Port  Status  Leader  Protocol  Build  Datacenter  Region
    nomad-server-01.global  192.168.0.20  4648  alive   false   2         1.0.4  dc1         global
    nomad-server-02.global  192.168.0.21  4648  alive   true    2         1.0.4  dc1         global
    nomad-server-03.global  192.168.0.22  4648  alive   false   2         1.0.4  dc1         global

Just one config change, and using SaltStack let us expand our Nomad server cluster from 1 to 3 and gives us redundancy.

Adding redundancy to Gluster

To add extra redundance to Gluster you'll need to take a couple of steps:

1. Make sure the Gluster nodes can find each other and can reach each other
2. Install the new Gluster nodes and peer them with the first
3. Create new bricks on the new nodes that we add to our existing storage volume
4. Make sure the volume is present in the future

Let's start with making sure the Gluster nodes can find each other (without using Consul Service discovery) by adding a new state file /srv/salt/gluster_hosts.sls:

{% from "config.jinja" import cluster with context %}

{% for name, ip in cluster.storage_server_mappings.items() %}
{{ name }} in /etc/hosts:
  host.present:
    - ip: {{ ip }}
    - names:
      - {{ name }}
{% endfor %}

And adding an extra parameter to our /srv/salt/config.jinja file:

{% set cluster = ({
    "subnet": "192.168.0.0/24",
    "ip_start": "192.168.0.",
    "consul_server_ips": ["192.168.0.10", "192.168.0.11", "192.168.0.12"],
    "storage_server_mappings": { "gluster-01": "192.168.0.100", "gluster-02": "192.168.0.101", "gluster-03": "192.168.0.102" },
}) %}

This makes sure that all our node knowledge stays centrally available in our cluster config array.

We'll need to add this state file to /srv/salt/top.sls:

<skipped everything in front>
  'storage-*':
    - match: pcre
    - consul_client
    - gluster_hosts
    - gluster_server

Before we install the new Gluster nodes, we require a new state in /srv/salt/gluster_server.sls after the disk mounting:

{% from "config.jinja" import cluster with context %}
{% set glusterfs_list = cluster.storage_server_mappings.keys() %}

storage brick1 mounted:
<SNIPPED>

peer-clusters:
  glusterfs.peered:
    - names:
{% for node in glusterfs_list %}
      - {{ node }}
{% endfor %}

storage volume present:
<SNIPPED>

Now let's have SaltStack install our Gluster nodes with the right state and check if they peer together by running:

$ sudo salt 'gluster-*' state.apply
$ sudo salt 'gluster-01' cmd.run "gluster peer status"
gluster-01:
    Number of Peers: 2

    Hostname: gluster-02
    Uuid: <SOME_UUID>
    State: Peer in Cluster (Connected)

    Hostname: gluster-03
    Uuid: <SOME_UUID>
    State: Peer in Cluster (Connected)

Great.. Our Gluster nodes can connect together and know about each of the other nodes.

Now we need to add the new Gluster bricks to gluster-02 and gluster-03 to fully replicate our existing storage volume over all three nodes.

$ sudo salt 'gluster-01' cmd.run "gluster volume add-brick storage replica 3 gluster-02:/data/gluster/storage/brick1/brick gluster-03:/data/gluster/storage/brick1/brick"
gluster-01:
    volume add-brick: success

Ok.. So now your volume has three more bricks. Let's check that this has been propagated:

$ sudo salt 'gluster-01' cmd.run "gluster volume status"
gluster-01:
    Status of volume: storage
    Gluster process                                          TCP Port  RDMA Port  Online  Pid
    ------------------------------------------------------------------------------
    Brick gluster-01:/data/gluster/storage/brick1/brick      49152     0          Y       89537
    Brick gluster-03:/data/gluster/storage/brick1/brick      49153     0          Y       14653
    Brick gluster-02:/data/gluster/storage/brick1/brick      49153     0          Y       14379
    Self-heal Daemon on localhost                            N/A       N/A        Y       14674
    Self-heal Daemon on gluster-02                           N/A       N/A        Y       89912
    Self-heal Daemon on gluster-03                           N/A       N/A        Y       14400

    Task Status of Volume storage
    ------------------------------------------------------------------------------
    There are no active volume tasks

You have now succeeded in expanding your storage cluster to fully redundant as well.

Adding redundancy to Docker

Adding redundancy to our Docker server is the easiest of all. We don't need to do anything other then install the Docker server with it's already configured state.

So just let SaltStack do what it does best:

$ sudo salt 'docker-*' state.apply

After this step finished we can check our Nomad node statuses:

$ sudo salt 'nomad-server-01' cmd.run "nomad node status"
nomad-server-01:
    ID        DC   Name       Class   Drain  Eligibility  Status
    38b945e8  dc1  docker-02  <none>  false  eligible     ready
    72a3cf21  dc1  docker-01  <none>  false  eligible     ready

Two docker nodes are available withing the cluster now. We are all done with redundancy.

This concludes the fifth part of our Avoiding the Cloud series. The sixth and final part will be about tightening down security of our cluster.