Multinode Zookeeper

22 Nov 2017

I've been learning Zookeeper, and figured I'd leave some notes on how I installed multinode Zookeeper.

Be sure you have jdk8 and zookeeper already installed as per Installing Zookeeper and Kafka

Let's be sure our zookeeper writes its transaction log to a different directory than the snapshots.

first-host$ cd ~/zookeeper-3.4.11/conf

first-host$ cat zoo.cfg 
tickTime=2000
dataDir=/home/mwood/zookeeper-3.4.11/data
clientPort=2181

first-host$ cat >> zoo.cfg 
dataLogDir=/home/mwood/zookeeper-3.4.11/xlog
^D

fist-host$ cat zoo.cfg 
tickTime=2000
dataDir=/home/mwood/zookeeper-3.4.11/data
clientPort=2181
dataLogDir=/home/mwood/zookeeper-3.4.11/xlog

first-host$ mkdir /home/mwood/zookeeper-3.4.11/xlog

Let's nuke zookeeper's current logs, and then start, and see what happens.

first-host$ cd /home/mwood/zookeeper-3.4.11/

first-host$ rm -rf data/*
first-host$ rm -rf xlog/*

first-host$ ./bin/zkServer.sh start

Now let's add some values and see what happens in the dirs.

first-host$ ./bin/zkCli.sh -server 127.0.0.1:2181

[zk: 127.0.0.1:2181(CONNECTED) 4] create /test_stuff my_data
Created /test_stuff
[zk: 127.0.0.1:2181(CONNECTED) 5] ls /
[zookeeper, test_stuff]
[zk: 127.0.0.1:2181(CONNECTED) 7] get /test_stuff
my_data
cZxid = 0x2
ctime = Wed Nov 15 12:39:19 EST 2017
mZxid = 0x2
mtime = Wed Nov 15 12:39:19 EST 2017
pZxid = 0x2
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 7
[zk: 127.0.0.1:2181(CONNECTED) 8] set /test_stuff my_changed_data
cZxid = 0x2
ctime = Wed Nov 15 12:39:19 EST 2017
mZxid = 0x3
mtime = Wed Nov 15 12:41:31 EST 2017
pZxid = 0x2
cversion = 0
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 15
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 9] get /test_stuff                
my_changed_data
cZxid = 0x2
ctime = Wed Nov 15 12:39:19 EST 2017
mZxid = 0x3
mtime = Wed Nov 15 12:41:31 EST 2017
pZxid = 0x2
cversion = 0
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 15
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 11] quit

Let's go look at our dirs:

first-host$ tree data
data
├── version-2
└── zookeeper_server.pid

1 directory, 1 file
first-host$ tree xlog/
xlog/
└── version-2
    └── log.1

That looks good!

OK, so now let's see if we can get zookeeper workin in a two node cluster.

first-host$ ./bin/zkServer.sh stop

Looks like we need to have the same config file on every server, and that every server gets a unique number to know which server it is.

There seems to be a config option, tick time, that is a multiplier in milliseconds. So let's make tick time 1000, so that one tick is one second, making the rest of the config easier to reason about.

tickTime=1000

Another config option, init limit, is how much time a non-leader zookeeper is allowed to take to connect to the leader. So let's try to limit this connection time to 10 seconds or less and see how that goes.

initLimit=10

Sync limit seems to be how out of date a non-leader can be with the leader. Some sample docs seem to set this at 4 seconds, which seems awfully generous, but let's try that.

syncLimit=4

Our cluster will have two servers, so apparently we need two server entries in our config file.

server.1=10.150.60.150:2888:3888
server.2=10.150.60.102:2888:3888

Our local server is known on the network as 10.150.60.150, and the other server I want to make work is 10.150.60.102: it does not yet have a JVM or Zookeeper installed on it yet. The two ports after each host are used by Zookeeper nodes to talk with each other. These come from the docs at https://zookeeper.apache.org/doc/r3.3.3/zookeeperStarted.html, so let's just go with that.

Let's actually put those configs in our config file (remember my zookeeper instance is stopped).

first-host$ cat conf/zoo.cfg 
tickTime=2000
dataDir=/home/mwood/zookeeper-3.4.11/data
clientPort=2181
dataLogDir=/home/mwood/zookeeper-3.4.11/xlog

first-host$ cat >> conf/zoo.cfg 
tickTime=1000
initLimit=10
syncLimit=4
server.1=10.150.60.150:2888:3888
server.2=10.150.60.102:2888:3888
^D

first-host$ cat conf/zoo.cfg 
tickTime=2000
dataDir=/home/mwood/zookeeper-3.4.11/data
clientPort=2181
dataLogDir=/home/mwood/zookeeper-3.4.11/xlog
tickTime=1000
initLimit=10
syncLimit=4
server.1=10.150.60.150:2888:3888
server.2=10.150.60.102:2888:3888

And now I guess I need to create the file that identifies this server as server 1.

first-host$ cat > data/myid
1
^D

Now let's ensure zookeeper is on our second host.

I won't show the exact commands, but I do these things:

So now I guess I just start zookeeper on both servers and see what happens!

New server:

other-host$ cat > data/myid 
2
^D

other-host$ ./bin/zkServer.sh start

Original server:

fist-host$ ./bin/zkServer.sh start

Original server:

first-host$ ./bin/zkCli.sh -server 127.0.0.1:2181
[zk: 127.0.0.1:2181(CONNECTED) 2] create /test_stuff my_data
[zk: 127.0.0.1:2181(CONNECTED) 4] ls /
[zookeeper, test_stuff]

New server:

other-host$ ./bin/zkCli.sh -server 127.0.0.1:2181
[zk: 127.0.0.1:2181(CONNECTED) 0] ls /
[zookeeper, test_stuff]
[zk: 127.0.0.1:2181(CONNECTED) 2] create /more_test_stuff more_data

Original server:

[zk: 127.0.0.1:2181(CONNECTED) 5] ls /
[zookeeper, more_test_stuff, test_stuff]

Nifty! Looks like we now have multinode Zookeeper!

Well, I guess now we should take down a zookeeper node and see what happens.

New server:

other-host$ ./bin/zkServer.sh stop

My CLI session on my old server seems to have started a failed attempt to keep reconnecting. Maybe because I started the new server first, so that was the leader? I hit ^C to stop that session. I will try to reconnect.

Nope, trying to restart the cli on the Original server still results in failed connection attempts. Interesting.

Let me restart the new instance.

OK, after I restart the new instance that I had stopped, then reconnecting with the cli from the Original server works again. Interesting! I would have thought some sort of failover would occur. Maybe I need to do more experimenting later.