Cluster Management Backends

Sheepdog uses a cluster management backend to manage membership and broadcast messages to the cluster nodes.

For now, sheepdog can use local driver (for development on a single box), corosync (the default), zookeeper and Accord.

Local Driver

This driver just makes use of Unix IPC mechanism to management the membership on a single box, where we start multiple 'sheep' processes to simulate the cluster. It is very easy and fast setup and especially useful to test functionality and for developer without involving any other software.

To set up a 3 node using local driver in one liner bash with debug mode (mkdir /path/to/store first):

$ for i in 0 1 2; do sheep -c local -d /path/to/store/$i -z $i -p 700$i;sleep 1;done

Corosync

Corosync runs on every node as a daemon.

Corosync can only work reliably with less than 15 nodes, due to its implementation and design goal for small sized cluster.

Zookeeper

Zookeeper runs as a standalone cluster. This means you need to set up zookeeper cluster first, then pass the IP:PORT list to sheep start-up option.

If you want to add more nodes into sheepdog cluster, you should run sheepdog against zookeeper. Some users (eg. at Taobao.com) have been working with the scalability of the sheepdog and currently running with around 1000 nodes in their test environment. For several month expediencies with zookeeper, they have found that it works well for node number below 1000, with object cache enabled.

To enable zookeeper support (Suppose we have a zookeeper cluster with 3 nodes):

Install zookeeper (by source tarball)

Fetch the newest tarball(>= 3.3.4)

$ wget http://mirror.bjtu.edu.cn/apache/zookeeper/zookeeper-3.3.4/zookeeper-3.3.4.tar.gz

Install zookeeper C client library(used by sheep)

$ tar -zxvf zookeeper-3.3.4.tar.gz
$ cd zookeeper-3.3.4/src/c
$ ./configure --prefix=/usr
$ make
$ make install

Set configuration(how to configure zookeeper)

$ cd zookeeper-3.3.4/conf
$ mv zoo_sample.cfg zoo.cfg
//Setting this to 0 entirely removes the limit on concurrent connections.
$ echo "maxClientCnxns=0" >> zoo.cfg
$ mkdir -p /tmp/zookeeper

Start zookeeper

$ sudo ./bin/zkServer.sh start 

Enable zookeeper driver in sheepdog

$ git clone git://github.com/collie/sheepdog.git
$ cd sheepdog/
$ ./configure --enable-zookeeper
$ make
$ sudo make install

Start sheepdog with zookeeper

$ sudo sheep -d /store/29 -z 29 -p 7029 -c zookeeper:127.0.0.1:2181

Note: We should not start multiple sheeps at the same time when use zookeeper driver. In fact, we just need to start the first sheep separately, after that, we can start other sheeps concurrently. Let's say, you want to start 100 sheeps:

- start the fist sheep alone, and sleep 2 seconds:
$ sheep -d /store/0 -z 0 -p 7000 -c zookeeper:localhost:2181
$ sleep 2

- start other sheeps simultaneously(need not to sleep between them):
$ for i in {1..99}; do sheep -d /store/$i -z $i -p $((7000 + $i)) -c zookeeper:localhost:2181

Install zookeeper (Debian-based distribution)

$ sudo apt-get install zookeeper zookeeperd

Start the zookeeper

$ /usr/share/zookeeper/bin/zkServer.sh start # The default port is 2181

or

$ /etc/init.d/zookeeper start

To compile sheep, firstly install zookeeper files (Debian-based distribution)

$ sudo apt-get install libzookeeper-dev

Then configure, make the sheep source

$ ./configure --enable-zookeeper
$ make
$ sudo make install

Start the sheep

$ sheep -c zookeeper:IP1:PORT1,IP2:PORT2,IP3:PORT3 ...other...option...

Note: We should not start multiple sheeps at the same time when use zookeeper driver. In fact, we just need to start the first sheep separately, after that, we can start other sheeps concurrently. Let's say, you want to start 100 sheeps:

- start the fist sheep alone, and sleep 2 seconds:
$ sheep -d /store/0 -z 0 -p 7000 -c zookeeper:localhost:2181
$ sleep 2

- start other sheeps simultaneously(need not to sleep between them):
$ for i in {1..99}; do sheep -d /store/$i -z $i -p $((7000 + $i)) -c zookeeper:localhost:2181

Accord

For a cluster more than 1000 nodes, I think Accord would come up to our rescue, but it is currently in a unstable development state. When the sheepdog scales up to 1000 nodes reliably, we might go to look at Accord and refine it to be a working state.