postgresql – How Much RAM for a Cloud-Hosted PostgreSQL Data Warehouse?

data-warehousememorypostgresql

I'm looking at migrating a current PostgreSQL data warehouse to a cloud host with SSD storage and RAM as one of the main sizing variables. The bulkiest data we're dealing with at the moment will live on monthly partitioned tables. Each month is about 70GB with indexes (40-ish w/o). Data is likely mostly bulk-loaded periodically, and will then be accessed by a small team of 5 researchers.

I've been trying to search for recommendations for spec'ing RAM on this site and all I've found is:

Fit the entire DB (>1TB, unrealistic)
More is better

Should there be enough RAM to at least load an entire index (16GB) to RAM? Are there any other details I should provide?

Best Answer

Estimating database server memory size by the size of tables (or indexes) is going to be misleading. On the one hand, depending on the database workload, it may not ever need to load the entire table (or index) in memory. On the other hand, there are many other possible memory consumers that have little or no correlation to the data size, such as sort buffers or client connections.

If your current server is not memory-bound, then you can use its configuration as a guideline. Otherwise you might as well buy yourself as much memory as you can afford.

Related Solutions

Postgresql – tuning postgresql for large amounts of ram

There are many tweakable constants, initialised via postgres.conf. The most important ones are:

max_connections: the number of concurrent sessions
work_mem : the maximal amount of memory to be used for intermediate results such as hash tables, and for sorting
shared_buffers the amount of memory dedicated to 'pinned' buffer space.
effective_cache_size the amount of memory assumed to be used by the OS's LRU buffers.
random_page_cost : an estimate for the relative cost of disk seeks.

max_connections should not be set higher than needed, connections cost resources even when idle; in most cases a connection would spend more time waiting inside than waiting outside. (at the price of concurrency) A nice rule-of-thumb formula is "number of spindles+number of processors+X"

work_mem is tricky: is can be applied to every subquery, so a query with 5 HASHJOINS might cost 5*work_mem. And for worst-case scenarios, you should also think of multiple sessions consuming this amount (again a reason to keep max_connections low).

shared_buffers is (IMHO) overrated. Normally it is advised to set it to about 1/4...1/2 of all available "free" memory, but I tend to keep it low, and set effective_cache_size to all available "free" memory.

random_page_cost is the cost for a seek+read on the disk. It is relative to the sequential_disk_cost, which is 1. The default (4) for random_page_cost is set too high for modern machines and network storage, normally it can be lowered to between 2 and 1.x. For SSD disks yould even set it to 1.0, since seeking is almost for free on SSDs.

Postgresql – How should I tune Postgresql for 20 GB of RAM

I would recommend using pgtune written by Greg Smith.

Simply run it on your server as follows:

pgtune -i postgresql.conf -o postgresql-tuned.conf

It has few more options, but just doing that and using generated postgresql.conf will do wonders for your server performance.

I think PostgreSQL should include this in standard install, and even run it by default - such that people cannot complain about bad PostgreSQL performance.

Best Answer

Related Solutions

Postgresql – tuning postgresql for large amounts of ram

Postgresql – How should I tune Postgresql for 20 GB of RAM

Related Question