Cassandra Consistency Notes (How to Invent Data)

13 Dec 2015

Launch all three nodes of a three-node cluster.

cqlsh> create keyspace foo with replication = {'class':'SimpleStrategy', 'replication_factor':3};
cqlsh> use foo;
cqlsh:foo> create table users (
    id int primary key,
    first_name text,
    last_name text);
cqlsh:foo> consistency all;
Consistency level set to ALL.
cqlsh:foo> insert into users (id, first_name, last_name) values (1, 'Manni', 'Wood');
cqlsh:foo> select * from users;

 id | first_name | last_name
----+------------+-----------
  1 |      Manni |      Wood

(1 rows)

Kill one of the other nodes; remember, we are still in consistency level ALL.

cqlsh:foo> insert into users (id, first_name, last_name) values (2, 'FirstOne', 'LastOne');
Traceback (most recent call last):
  File "./cqlsh.py", line 1216, in perform_simple_statement
    result = future.result()
  File "/home/mwood/apache-cassandra-3.0.0/bin/../lib/cassandra-driver-internal-only-3.0.0a3.post0-3f15725.zip/cassandra-driver-3.0.0a3.post0-3f15725/cassandra/cluster.py", line 3118, in result
    raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level ALL" info={'required_replicas': 3, 'alive_replicas': 2, 'consistency': 'ALL'}

Very nice!

Bring the dead node back up

cqlsh:foo> insert into users (id, first_name, last_name) values (2, 'FirstOne', 'LastOne');

Insert now works.

cqlsh:foo> select writetime(first_name), first_name from users;

 writetime(first_name) | first_name
-----------------------+------------
      1449548571402412 |      Manni
      1449548965888756 |   FirstOne

(2 rows)

Take down all but node one

cqlsh:foo> insert into users (id, first_name, last_name) values (3, 'AAA', 'ZZZ') using timestamp 1449548965888760;
Traceback (most recent call last):
  File "./cqlsh.py", line 1216, in perform_simple_statement
    result = future.result()
  File "/home/mwood/apache-cassandra-3.0.0/bin/../lib/cassandra-driver-internal-only-3.0.0a3.post0-3f15725.zip/cassandra-driver-3.0.0a3.post0-3f15725/cassandra/cluster.py", line 3118, in result
    raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level ALL" info={'required_replicas': 3, 'alive_replicas': 1, 'consistency': 'ALL'}

cqlsh:foo> consistency quorum;
Consistency level set to QUORUM.

cqlsh:foo> insert into users (id, first_name, last_name) values (3, 'AAA', 'ZZZ') using timestamp 1449548965888760;
Traceback (most recent call last):
  File "./cqlsh.py", line 1216, in perform_simple_statement
    result = future.result()
  File "/home/mwood/apache-cassandra-3.0.0/bin/../lib/cassandra-driver-internal-only-3.0.0a3.post0-3f15725.zip/cassandra-driver-3.0.0a3.post0-3f15725/cassandra/cluster.py", line 3118, in result
    raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={'required_replicas': 2, 'alive_replicas': 1, 'consistency': 'QUORUM'}

cqlsh:foo> consistency one;
Consistency level set to ONE.

cqlsh:foo> insert into users (id, first_name, last_name) values (3, 'AAA', 'ZZZ') using timestamp 1449548965888760;

In it goes!

Turn off node 1. Now all nodes are off.

cqlsh:foo> insert into users (id, first_name, last_name) values (3, 'AAA', 'ZZZ') using timestamp 1449548965888760;
NoHostAvailable: ('Unable to complete the operation against any hosts', {})

Oh, good.

Turn on only node 2

$ ./cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})

Oh, I see.

$ ./cqlsh 127.0.0.2
Connected to Test Cluster at 127.0.0.2:9042.
[cqlsh 5.0.1 | Cassandra 3.0.0 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.

better

cqlsh> use foo;
cqlsh:foo> 

cqlsh:foo> consistency one;
Consistency level set to ONE.

cqlsh:foo> insert into users (id, first_name, last_name) values (3, 'BBB', 'YYY') using timestamp 1449548965888760;

And in it goes!

OK, let's turn off node 2, and then turn on only node 3.

$ ./cqlsh 127.0.0.3
Connected to Test Cluster at 127.0.0.3:9042.
[cqlsh 5.0.1 | Cassandra 3.0.0 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> use foo;
cqlsh:foo> consistency one;
Consistency level set to ONE.

cqlsh:foo> insert into users (id, first_name, last_name) values (3, 'CCC', 'XXX') using timestamp 1449548965888760;

In that goes!

Now let's bring up the other two nodes.

Let's stay with our connection to node 3

cqlsh:foo> consistency;
Current consistency level is ONE.

cqlsh:foo> select * from users where id = 3;

 id | first_name | last_name
----+------------+-----------
  3 |        CCC |       ZZZ

(1 rows)

Wow. OK, so repairs (or whatever) have already happened, and the two lexicograpically higher columns were chosen. There was no user CCC ZZZ, but now there is!

If we updated, could we protect against this by including the columns in our update's where clause?

cqlsh:foo> update users set first_name = 'DDD', last_name = 'WWW' where id = 3 and first_name = 'AAA' and last_name = 'ZZZ';
InvalidRequest: code=2200 [Invalid query] message="Non PRIMARY KEY columns found in where clause: first_name, last_name "

Nope! There is basically no way to prevent against the "lost update" problem, let alone this sort of corruption.