Postgresql crash in master node with streaming replication

crashpostgresqlreplication

We have a master/slave postgresql cluster with streaming replication and pgpool.

Versions in both postgresql servers are.

postgres: 9.4
OS: Debian GNU/Linux 8 (jessie)

PGPOOL server:
version pgpool: pgpool-II version 3.3.3
os: Red Hat Enterprise Linux Server release 6.1

Problem:

We suffer a crash down of the postgresql service at master node with subsequent failover done by pgpool

Could not find the reason of this behavior in master node. We want to know if someone coult tell us, based on the recorded log messages, what could have caused this crash.
Following, some relevant lines from the postgresql and linux log.

postgresql.log

2020-12-20 21:06:04 UYT [25922]: [4-1] user=,db=,app=,client= LOG:  server process (PID 19303) was terminated by signal 11: Segmentation fault
2020-12-20 21:06:04 UYT [25922]: [5-1] user=,db=,app=,client= DETAIL:  Failed process was running: COPY public.act_hi_varinst (id_, proc_def_key_, proc_def_id_, proc_inst_id_, execution_id_, act_inst_id_, case_def_key_, case_def_id_, case_inst_id_, case_execution_id_, task_id_, name_, var_type_, rev_, bytearray_id_, double_, long_, text_, text2_, tenant_id_, state_, create_time_, root_proc_inst_id_, removal_time_) TO stdout;
2020-12-20 21:06:04 UYT [25922]: [6-1] user=,db=,app=,client= LOG:  terminating any other active server processes
2020-12-20 21:06:04 UYT [1910]: [3-1] user=dbusr,db=dbdatabase     ,app=[unknown],client= WARNING:  terminating connection because of crash of another server process
2020-12-20 21:06:04 UYT [1910]: [4-1] user=dbusr,db=dbdatabase     ,app=[unknown],client= DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-12-20 21:06:04 UYT [1910]: [5-1] user=dbusr,db=dbdatabase     ,app=[unknown],client= HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2020-12-20 21:06:04 UYT [27108]: [3-1] user=dbusr,db=dbdatabase     ,app=[unknown],client= WARNING:  terminating connection because of crash of another server process
2020-12-20 21:06:04 UYT [27108]: [4-1] user=dbusr,db=dbdatabase     ,app=[unknown],client= DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

Error found at /var/log/messages

Dec 20 21:06:04 server-name kernel: [20278231.774013] postgres[19303]: segfault at 7f7740354704 ip 00007f772c8ac700 sp 00007ffdac7ff4e0 error 4 in postgres[7f772c813000+5a1000]

Thank you all.

Best Answer

If that crash is repeatable, you probably have data corruption in your table. If the crash is not repeatable, the cause could be faulty memory.

At any rate, the odds are good that your hardware has problems. That would not surprise me, if you are as negligent with hardware maintenance as you are with software maintenance (9.4 is out of support).

Use reliable server grade hardware and keep your software versions reasonably recent.

Related Question