Build a 3-node mongodb cluster using puppet (for use with High Availability Graylog in this case)

One of the core components to a Graylog installation in MongoDB. Quite possibly the worst database ever to grace the planet :)

Hopefully, from a Graylog prospective, MongoDB will disappear from the solution soon.

Anyway, from an architecture prospective, we want to use a highly available Graylog deployment aka Graylog HA. Of which, there is little documentation about. So from a technical prospective you’ve got:

  • Incoming log traffic load-balancer
  • Multiple Graylog servers
  • Multiple MongoDB nodes (also Graylog servers)
  • Multiple ElasticSearch nodes

In our case, we chose to use:

  • A NetScaler to listen on UDP 514 and also host the SSL certificate.
    • The NetScaler will also do a API call against the Graylog servers to verify health.
    • The NetScaler will then pass the traffic to the active Graylog server on the active input thats listening on UDP 5140.
  • The two Graylog servers will be part of a MongoDB cluster, and then a third VM will be used as a MongoDB witness server.
  • Three servers will be used a ElasticSearch nodes.

From a configuration management prospective, we wanted to leverage Puppet to do the installation of the MongoDB cluster.

The puppet manifests we used are:

Using haproxy as a load balancer for OpenStack services on RedHat OpenStack

Configuring RedHat OpenStack to be highly available is not like VMware where you just enable a couple features, check some boxes and voila! Quite the contrary… In fact; configuring RedHat OpenStack to be highly available is quite elegant.

Lets look at it like this. You quite a few services that make up OpenStack as a product. Services like Keystone (auth), Neutron (networking), Glance (image storage), Cinder (volume storage) and Nova (compute/scheduler) and MySQL (database).  Some of these services are made highly available via failover and some via cloning; aka having multiple copies of the same service distributed throughout the deployment.

OpenStack HA
OpenStack HA

In this case, we need to configure the clone pair of load balancers for to use for our OpenStack services. This will allow us to reference a single Virtual IP that will be load balanced across all cloned service nodes. For this functionality, were going to use haproxy for our load balancing software and pacemaker for clustering.

Load Balanced OpenStack Services
Load Balanced OpenStack Services

The first step is going to be to configure the RHEL HA and LB yum channels with the following:

Then simply install the HA and LB packages with:

The next step being, configure /etc/haproxy/haproxy.cfg to look like this on both of your load balancer nodes:

The next step is to then configure pacemaker on each of the nodes to start haproxy accordingly after the resources are configured.

The next step is to define the cluster resources

Now the only thing left to do is to create the stonith resources

You should now have haproxy running as a clustered service servicing a Virtual IP address for your OpenStack Services.