Postgresql – Postgres backup and WAL to S3

In order to restore a backup, you need to have the base archive of all the data files, plus a sequence of xlogs. An "incremental backup" can be made, of just some more xlogs in the sequence. Note that if you have any missing xlogs, then recovery will stop early.

So it's not clear here exactly what you've done, unless you changed the level of detail you're mentioning part way through your list. When you make a copy of more segments that have been put into the archive directory after adding more data, you need to ensure that all the data has been archived: using pg_start_backup and pg_stop_backup usually does this for you, but you don't mention it the second time. You need to at least do a pg_switch_xlog to have the current xlog segment immediately archived.

If you think that recovery is not consuming enough xlog segments, look at the recovery log to see if it tried to take them all. And have your recovery command make some sort of mark on which xlog files were taken.

The error message FATAL: the database system is shutting down is probably due to new incoming connections that are denied because of the shutdown state of the master process.

According to Shutting Down the Server in the documentation, for a fast shutdown, the existing server processes have received a SIGTERM signal to ask them to terminate as soon as possible.

Yet there is this process:

postgres 14306 14266  9 May28 ?        04:01:55 postgres: opera_man om 10.40.227.146(44745) SELECT

that does not want to quit.

You may try the next option, sending SIGQUIT to the master process (pid 14266). This is likely to make it quit and it remains to be seen whether the subprocess with pid 14306 will quickly abort or stay stuck.

It it's stuck, I would kill it with SIGKILL as the last resort and then restart postgres normally with the init script.