Cassandra in a Public Cloud – Which Snitch

In a company I work for we are dealing with similar amount of data (around 10 TBs of realtime searchable data). We solve this with Cassandra and I would like to mention couple of ideas that will allow you to do O(1) search on a multi TBs database. This is not specific to Cassandra db though, you can use it with other db as well.

Theory

• Shard your data. There is no way a single server will reliably and realistically hold such volume of data.
• Be ready for hardware faults and whole node failures, duplicate the data.
• Start using many back-end servers from the beginning.
• Use many cheaper commodity servers, compared to top-end high performance ones.
• Make sure data is equally distributed across shards.
• Spend a lot of time planning your queries. Derive API from the queries and then carefully design tables. This is the most important and prolonged task.
• In Cassandra, you can design a composite column key and get access to that key in O(1). Spend time working on them. This will be used to access searchable records instead secondary index.
• Make use of wide rows. They are useful for storing time-stamped events.
• Never perform full-scan or in fact any operation more than O(Log N) on such volume. If you require anything more than O(Log N), offload such operations to Map-Reduce algorithms.

Practice

• Don't spend time building OS images or installing servers on physical machines. Use cloud based providers for quick prototyping. I worked with Amazon EC2 and can highly recommend it for its simplicity, reliability and speed of prototyping.
• Windows machines tend to be slower during boot time and take considerably more resources being in Idle state. Consider using Unix-based OS. Personally, I found Ubuntu server to be a reliable OS, but moreover there is a pretty good community at askubuntu
• Think about networking, nodes shall ideally be close to each other to allow fast gossiping and meta-data exchange.
• Do not go into extreme cases: really wide column rows or exceptionally long column families (tables). Best performance is achieved in the sane boundaries - if db supports that many N rows by design, it doesn't mean it performs well.
• Our search takes about 3-5 seconds, much is due to the intermediate nodes between UI and the database. Consider how to bring requests closer to the database.
• Use a network load balancer. Choose an established one. We use HAProxy, which is simple, but dead fast. Never had problems with it.
• Prefer simplicity to complex solutions.
• Look for free open-source solutions, unless you are backed up by a corporation's size budget. Once you go more than several servers, the costs of infrastructure might go sky high.

I do not work for Amazon and have no relations with HAProxy and Ubuntu teams. This is a personal opinion rather than any sort of promotion.

Memtable flush happens in 3 ways
1. When memtable_offheap_space_in_mb size is exceeded.
2. When commitlog_total_space_in_mb size is exceeded.
3.When forcefully Nodetool flush commend is executed.
In your case I'm guess for second reason that flush happened. For more you can check this. http://academy.datastax.com/courses/understanding-cassandra-write-path/understanding-memtables-and-commitlog