What may be the consequences of deleting the system keyspace

cassandraclusteringnosql

I have a Cassandra cluster composed of 2 nodes. One node could not start Cassandra yesterday. I deleted the system keyspace directory and I started the node again. The node is running after system keyspace directory deletion.

What are consequences of system keyspace missing?

All data are present in my cluster and I find no errors at the moment.

Best Answer

The system keyspace is where a Cassandra node maintains information about itself and other nodes. So things like network addresses, responsible token ranges, status from gossip, hints, schema, and other local information. The system keyspace has its own, special replication strategy known as "LocalStrategy:"

cqlsh:system> desc KEYSPACE;

CREATE KEYSPACE system WITH replication = {
  'class': 'LocalStrategy'
};

This should tell you that data in the system keyspace isn't replicated to other nodes, and is largely specific to itself.

Sometimes you can fix gossip-related issues by blowing-away the system keyspace (or even just all entries in system.peers). Removing it via the filesystem (as you did) has the same effect. The reason this works, is because it forces the node to rewrite all of that data, most of which it already has (system.local) or it can learn from gossip.

While I wouldn't make a habit out of it, deleting the system keyspace via the filesystem is ok...as long as you have another node in the cluster for it to communicate with.

Related Solutions

Mariadb – The right way of starting up a Galera Cluster

At MariaDB 10.1 where Galera included you need to start first node with key --wsrep-new-cluster

e.g. /etc/init.d/mysql start --wsrep-new-cluster

All other nodes starts as usually:

/etc/init.d/mysql start

To determine which node needs to be bootstrapped, compare the wsrep_last_committed value on all DB nodes:

KVM-1> SHOW STATUS LIKE 'wsrep_%';
+----------------------+-------------+
| Variable_name        | Value       |
+----------------------+-------------+
| wsrep_last_committed | 21568       |
...
| wsrep_cluster_status | non-Primary |
+----------------------+-------------+

KVM-2> SHOW STATUS LIKE 'wsrep_%';
+----------------------+-------------+
| Variable_name        | Value       |
+----------------------+-------------+
| wsrep_last_committed |  1359       |
...
| wsrep_cluster_status | non-Primary |
+----------------------+-------------+



KVM-3> SHOW STATUS LIKE 'wsrep_%';
+----------------------+-------------+
| Variable_name        | Value       |
+----------------------+-------------+
| wsrep_last_committed |   537       |
...
| wsrep_cluster_status | non-Primary |
+----------------------+-------------+

KVM-1 has the most up-to-date data. In this case, all Galera nodes are already started, so you don’t necessarily need to bootstrap the cluster again. We just need to promote KVM-1 to be a Primary Component:

KVM-1> SET GLOBAL wsrep_provider_options="pc.bootstrap=1";

The remaining nodes will then reconnect to the Primary Component (KVM-1) and resyncing back data based on this node.

Cassandra Clustering – Is it Safe to Add a New Node During Repair?

As per Datastax documentation,

Warning: DataStax recommends stopping repair operations during topology changes; the Repair Service does this automatically. Repairs running during a topology change are likely to error when it involves moving ranges.

So, it is clear that topology changes (adding/removing nodes) are not recommended when there is a repair process alive in your cluster

Best Answer

Related Solutions

Mariadb – The right way of starting up a Galera Cluster

Cassandra Clustering – Is it Safe to Add a New Node During Repair?

Related Question