The message "The database system is starting up." does not indicate an error. The reason it is at the FATAL level is so that it will always make it to the log, regardless of the setting of log_min_messages:
http://www.postgresql.org/docs/9.1/interactive/runtime-config-logging.html#RUNTIME-CONFIG-LOGGING-WHEN
After the rsync, did you really run what you show?:
pgsql -c "select pg_stop_backup();";
Since there is, so far as I know, no pgsql executable, that would leave the backup uncompleted, and the slave would never come out of recovery mode. On the other hand, maybe you really did run psql, because otherwise I don't see how the slave would have logged such success messages as:
Log: consistent recovery state reached at 0/BF0000B0
and:
Log: streaming replication successfully connected to primary
Did you try connecting to the slave at this point? What happened?
The "Success. You can now start..." message you mention is generated by initdb, which shouldn't be run as part of setting up a slave; so I think you may be confused about something there. I'm also concerned about these apparently conflicting statements:
The only ways I have restarted Postgres is through the service
postgresql-9.1 restart or /etc/init.d/postgresql-9.1 restart commands.
After I receive this error, I kill all processes and again try to
restart the database...
Did you try to stop the service through the service script? What happened? It might help in understanding the logs if you prefixed lines with more information. We use:
log_line_prefix = '[%m] %p %q<%u %d %r> '
The recovery.conf script looks odd. Are you copying from the master's pg_xlog directory, the slave's active pg_xlog directory, or an archive directory?
Based on your situation, it would make sense I think to disable whatever PostgreSQL you have and start over with a fresh install. We are dealing with completely different applications that have nothing to do with eachother. In reality, PostgreSQL is only needed to serve up data. It makes sense to me, to simplify your life by installing PostgreSQL seperately and treating it as a totally independant application.
The simplest way to go about this, would be to take the following steps.
- Determine if PostgreSQL is currently running
A. If it is, locate it and disable it
B. If it isn't move onto step 2
- Download PostgreSQL 9.2 from PostgreSQL's website
- Install dependancies
- Configure, compile and install postgresql
- Initalize your database and start postgres
- What's next
1. Determine if PostgreSQL is currently running
From the shell, type the following
[root@someserver ~]# netstat -tlpn | grep postmaster
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 1119/postmaster
tcp 0 0 :::5432 :::* LISTEN 1119/postmaster
Postmaster on my machine is the PostgreSQL service. It is listening on port 5432 and has a process id of 1119.
Where is it?
Now that we have the process ID we can locate the files being used. lsof is handy because it will show you all the files and sockets a given process is using.
[root@someserver ~]# lsof -p 1119
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postmaste 1119 postgres cwd DIR 253,3 4096 131093 /var/lib/pgsql/9.1/data
postmaste 1119 postgres rtd DIR 253,0 4096 2 /
postmaste 1119 postgres txt REG 253,4 5205008 27145 /usr/pgsql-9.1/bin/postgres
On my server, it is using /usr/pgsql-9.1/bin/postgres to run the service and the /var/lib/pgsql/9.1/data is the data directory.
From this point you know where its running and what directories its using.
On your server, it looks like its installed in /usr/local/Cellar/postgresql. If you have to start it manually, then you can just leave it alone. When you install PostgreSQL from source, it will install in /usr/local/pgsql by default.
It is only important to be sure that the postgresql currently installed on your system is not using port 5432. This is because two services can not use the same port. If it is, you will need to kill the process and keep it from starting again. Based the contents of your question, it looks like that won't be a problem.
2. Download postgresql
From the shell
[root@someserver ~]# cd ~
[root@someserver ~]# wget http://ftp.postgresql.org/pub/source/v9.2.2/postgresql-9.2.2.tar.gz
[root@someserver ~]# tar -zxvf postgresql-9.2.2.tar.gz
3. Install dependencies
I didn't notice you mentionning what flavor of Linux you were running. You can identify your OS by running cat /etc/issue.
PostgreSQL depends on several libraries.
On CentOS, I would run this to install the libraries I needed
[root@someserver ~]# yum install readline-devel zlib-devel make gcc g++
On Debian, you would use apt-get to install and apt-cache search to locate. I included build-essential because you'll need it to compile postgres.
Something like apt-get install libreadline6 libreadline6-dev build-essential
Not sure about OSX though
4. Configure, Compile and Install PostgreSQL
[root@someserver ~]#
cd postgresql-9.2.2
The --bindir option will put the the postgresql binaries like pg_ctl, initdb, createdb all in the /usr/bin directory. This will be handy for you since you mentionned having trouble finding them.
./configure --bindir=/usr/bin
make
make install
You can check to see if the binaries are available
[root@someserver ~]# psql -V
psql (PostgreSQL) 9.2.2
Once this is done, normally you would need to add the postgres user, but he's already on your system
You do however need to create a data directory and give postgres read/write access to it
[root@someserver ~]# mkdir -p /var/lib/pgsql/data
[root@someserver ~]# chown postgres:postgres -R /var/lib/pgsql/data
[root@someserver ~]# su - postgres
PostgreSQL is installed
You have created a data directory
Now you can initalize your database
5. Initalize and start postgres
initdb -D /var/lib/pgsql/data
This will the cluster and your configuration files such as postgresql.conf, pg_hba.conf which you will need to modify later based on the users that need access, their ip addresses and what ip address and port postgresql will listen on
Now you can start PostgreSQL with the following command :
/usr/local/pgsql/bin/postmaster -D /var/lib/pgsql/data > /var/lib/pgsql/logfile 2>&1 &
When you run netstat -tlpn again, you should see
[root@someserver ~]# netstat -tlpn | grep postmaster
tcp 0 0 127.0.0.1:5432 127.0.0.1:* LISTEN 8721/postmaster
It is only listening to local connections for now, because by defaut that's what postgresql.conf is configured to do.
6. What's next
From here, you would need to configure postgresql.conf and pg_hba.conf appropriately. pg_hba.conf defines who can connect, from where and to what. postgresql.conf is used to configure the entire service, what ip address or addresses it listens on, what port number, maintenance cyles, etc. For your purposes, you will probabaly only be interested in changing this variable
listen_addresses = '*'
You may also want to create an init script in the /etc/init.d directory, so it will start automatically. How you go about that also depends on what flavor of Linux you are running.
If you run into any problems let me know. Since you are probably not using the exact same system, you may run into problems I haven't discussed here.
Best Answer
Just use
tail -f /path/to/logand you'll be able to see the log contents in a terminal as well. You can use a terminal tool that can "monitor for activity" if you'd like to be notified when something happens.A more complex solution would be to set PostgreSQL to log to rsyslog, and set rsyslog to log to a tty as well as a file