The message "The database system is starting up." does not indicate an error. The reason it is at the FATAL level is so that it will always make it to the log, regardless of the setting of log_min_messages:
http://www.postgresql.org/docs/9.1/interactive/runtime-config-logging.html#RUNTIME-CONFIG-LOGGING-WHEN
After the rsync, did you really run what you show?:
pgsql -c "select pg_stop_backup();";
Since there is, so far as I know, no pgsql executable, that would leave the backup uncompleted, and the slave would never come out of recovery mode. On the other hand, maybe you really did run psql, because otherwise I don't see how the slave would have logged such success messages as:
Log: consistent recovery state reached at 0/BF0000B0
and:
Log: streaming replication successfully connected to primary
Did you try connecting to the slave at this point? What happened?
The "Success. You can now start..." message you mention is generated by initdb, which shouldn't be run as part of setting up a slave; so I think you may be confused about something there. I'm also concerned about these apparently conflicting statements:
The only ways I have restarted Postgres is through the service
postgresql-9.1 restart or /etc/init.d/postgresql-9.1 restart commands.
After I receive this error, I kill all processes and again try to
restart the database...
Did you try to stop the service through the service script? What happened? It might help in understanding the logs if you prefixed lines with more information. We use:
log_line_prefix = '[%m] %p %q<%u %d %r> '
The recovery.conf script looks odd. Are you copying from the master's pg_xlog directory, the slave's active pg_xlog directory, or an archive directory?
Without knowing your specific errors, I dont know what the easiest approach to recovering your database is. For example, there could be minor problems like bad settings that cause PostgreSQL to be unable to start. I would recommend asking a new question about getting PostgreSQL to start before looking at how to recover your data.
However, I can answer what are some aspects of your question about recovering data. I have actually never seen data corruption in PostgreSQL, and so all my experience with it is second-hand (email list threads and the like). It is very rare, and usually related to failing hardware. If you do have data corruption, my recommendation is to restore from backup onto a new server while you run diagnostics and determine the problem on your existing server.
When you talk about recovering from the files in pg_xlog you are talking about using write logs as incremental backups. You still need a compatible starting point, which means effectively you are going with a base backup plus all pg_xlog segments created from that point onward.
Now, if you have no good backups, and you have a lot of important data, it may be possible to hire an expert for significant expense to recover some or all of your data. This is not cheap.
I think your immediate steps are to step back and ask why PostgreSQL is not starting and see what you can do about that first before assuming you need to recover your data. If you need to recover your data, you hopefully have backups.
Best Answer
To find out where the PostgreSQL data folder is located, just run SQL query:
You can find more details here
However, note
File System Level Backup are faster but large in size. The worst thing is that to perform a database's backup, PostgreSQL Server required to be stopped. More information about the restrictions of this operation can be found in the official documentation. On my own I will add that from such backups, as a rule, there is more harm than good.
The main method for creating backups in PostgreSQL is the pg_dump utility.
Opposite file-level backup, a backup via the pg_dump tool does not require stopping the server, it is easy to automate and does not require access to the server file system (it can be performed remotely).