Cassandra Performance – Writing Large Partitions

I initially just wanted to point at this as a comment, but my rep is too low. So excuse me if this is not a fully fledged answer. (Anyone feel free to edit or hint in comments)

Looking at the documentation at datastax it appears that data will be evently divided by nodes, depending on the "automagical" tokens. From my understanding it's simple as that a new vnode with new tokens just takes an even portion of each node, like all the other nodes already do. Hell, this is even what the doc says basically word by word:

Rebalancing a cluster is no longer necessary when adding or removing nodes. When a node joins the cluster, it assumes responsibility for an even portion of data from the other nodes in the cluster. If a node fails, the load is spread evenly across other nodes in the cluster.

And to address your question:

To further specify my question for virtual nodes does the cluster re-evaluate tokens for all nodes just because a new host has been added?

Sort of. The new node itself takes some partitions (evenly) from all other nodes. If you remove a node every other node will rebalance its partitions from the other nodes, making up for the lost node.

Schema design in Cassandra, for efficient tables, will grate against your RDBMS experience; for efficiency, the Cassandra prefers denormalization, not normalization. By this, I mean that if you have some user information and you want to look up that data using two different primary keys, then using Cassandra, it actually is better to use two tables (and duplicate the data). Yes, this means more storage space, but it also allows for faster reads.

As a side note, based on my own experiences, I would recommend against using a secondary index, and instead simply use another table. Secondary indexes in Cassandra are treated a little differently, with background threads which update the indexes periodically; this makes reading from an index not quite as reliable (i.e. more likely to surprise you, in a not good way) than just using a table.

Thus I would recommend something like the following two tables for your needs:

CREATE TABLE users (
  id TIMEUUID PRIMARY KEY,
  user UUID,
  friend UUID
);

CREATE TABLE friends (
  id TIMEUUID,
  user UUID,
  friend UUID,
  PRIMARY KEY (user, friend)
);

This second table would let you do your CQL query:

SELECT * FROM friends WHERE user = ?

Notice that this friends table uses a compound primary key. This allows there to be multiple friend values associated with that single user value.

One of the downsides of this multiple-table approach is that your application code now has to be responsible for writing into both tables for a single "update", and you have to deal with any potential skew/reconciliation. Cassandra achieves its performance, in many ways, by avoiding enforcing of foreign key constraints and such and leaving that to the application.

Hope this helps!