Redis – High RSS Memory After Lowering maxmemory

1. Use the maxmemory to set a limit to how much your Redis database can grow too. Failing to do so, Redis will grow until the OS will kill it once memory is exhausted (per your current experience).
2. The usage of maxmemory should be coupled with maxmemory-policy - you can choose from different eviction policies depending on your use case's requirements. For example, if you use the allkeys-lru eviction policy, Redis will indeed start evicting (the least recently used) data once maxmemory has been reached. Alternatively, you can instruct Redis to evict only expire-able data with the volatile-lru or volatile-random policies. Lastly, you can set the policy to noeviction but that would mean that once memory has been exhausted, Redis will deny further writes with an OOM message.

Edit:

First disable swap - Redis and swap don't mix easily and this can certainly cause slowness.

Also do free -m instead of top for the full picture of your RAM's state (http://www.linuxatemyram.com/).