Galera Cluster – Troubleshooting Replication Issues

galeramariadbMySQLreplication

I am using MariaDB 10.2

I have node2 (2.2.2.2) running as a slave to some external database. I bootstrapped node2 as the first member of a Galera cluster "my_cluster".

Node1 (1.1.1.1) joined with rsync SST but after an hour I see that it's behind node2.

I tried running some inserts manually and it works both ways.
However anything added by the replication to node2 in completely ignored by node1.

node1:

MariaDB [db1]> SHOW STATUS LIKE 'wsrep%';
+------------------------------+--------------------------------------+
| Variable_name                | Value                                |
+------------------------------+--------------------------------------+
| wsrep_apply_oooe             | 0.000000                             |
| wsrep_apply_oool             | 0.000000                             |
| wsrep_apply_window           | 0.000000                             |
| wsrep_causal_reads           | 0                                    |
| wsrep_cert_deps_distance     | 0.000000                             |
| wsrep_cert_index_size        | 0                                    |
| wsrep_cert_interval          | 0.000000                             |
| wsrep_cluster_conf_id        | 6                                    |
| wsrep_cluster_size           | 2                                    |
| wsrep_cluster_state_uuid     | 09e3b6c8-343c-11e8-87cf-07a9813fdf95 |
| wsrep_cluster_status         | Primary                              |
| wsrep_commit_oooe            | 0.000000                             |
| wsrep_commit_oool            | 0.000000                             |
| wsrep_commit_window          | 0.000000                             |
| wsrep_connected              | ON                                   |
| wsrep_desync_count           | 0                                    |
| wsrep_evs_delayed            |                                      |
| wsrep_evs_evict_list         |                                      |
| wsrep_evs_repl_latency       | 0/0/0/0/0                            |
| wsrep_evs_state              | OPERATIONAL                          |
| wsrep_flow_control_paused    | 0.000000                             |
| wsrep_flow_control_paused_ns | 0                                    |
| wsrep_flow_control_recv      | 0                                    |
| wsrep_flow_control_sent      | 0                                    |
| wsrep_gcomm_uuid             | 8854b393-3713-11e8-8cfd-f7a101a4c6bf |
| wsrep_incoming_addresses     | 1.1.1.1:3306,2.2.2.2:3306            |
| wsrep_last_committed         | 0                                    |
| wsrep_local_bf_aborts        | 0                                    |
| wsrep_local_cached_downto    | 18446744073709551615                 |
| wsrep_local_cert_failures    | 0                                    |
| wsrep_local_commits          | 0                                    |
| wsrep_local_index            | 0                                    |
| wsrep_local_recv_queue       | 0                                    |
| wsrep_local_recv_queue_avg   | 0.000000                             |
| wsrep_local_recv_queue_max   | 1                                    |
| wsrep_local_recv_queue_min   | 0                                    |
| wsrep_local_replays          | 0                                    |
| wsrep_local_send_queue       | 0                                    |
| wsrep_local_send_queue_avg   | 0.000000                             |
| wsrep_local_send_queue_max   | 1                                    |
| wsrep_local_send_queue_min   | 0                                    |
| wsrep_local_state            | 4                                    |
| wsrep_local_state_comment    | Synced                               |
| wsrep_local_state_uuid       | 09e3b6c8-343c-11e8-87cf-07a9813fdf95 |
| wsrep_protocol_version       | 8                                    |
| wsrep_provider_name          | Galera                               |
| wsrep_provider_vendor        | Codership Oy <info@codership.com>    |
| wsrep_provider_version       | 25.3.23(r3789)                       |
| wsrep_ready                  | ON                                   |
| wsrep_received               | 3                                    |
| wsrep_received_bytes         | 219                                  |
| wsrep_repl_data_bytes        | 0                                    |
| wsrep_repl_keys              | 0                                    |
| wsrep_repl_keys_bytes        | 0                                    |
| wsrep_repl_other_bytes       | 0                                    |
| wsrep_replicated             | 0                                    |
| wsrep_replicated_bytes       | 0                                    |
| wsrep_thread_count           | 2                                    |
+------------------------------+--------------------------------------+

node2:

MariaDB [db1]> SHOW STATUS LIKE 'wsrep%';
+------------------------------+--------------------------------------+
| Variable_name                | Value                                |
+------------------------------+--------------------------------------+
| wsrep_apply_oooe             | 0.000000                             |
| wsrep_apply_oool             | 0.000000                             |
| wsrep_apply_window           | 0.000000                             |
| wsrep_causal_reads           | 0                                    |
| wsrep_cert_deps_distance     | 0.000000                             |
| wsrep_cert_index_size        | 0                                    |
| wsrep_cert_interval          | 0.000000                             |
| wsrep_cluster_conf_id        | 6                                    |
| wsrep_cluster_size           | 2                                    |
| wsrep_cluster_state_uuid     | 09e3b6c8-343c-11e8-87cf-07a9813fdf95 |
| wsrep_cluster_status         | Primary                              |
| wsrep_commit_oooe            | 0.000000                             |
| wsrep_commit_oool            | 0.000000                             |
| wsrep_commit_window          | 0.000000                             |
| wsrep_connected              | ON                                   |
| wsrep_desync_count           | 0                                    |
| wsrep_evs_delayed            |                                      |
| wsrep_evs_evict_list         |                                      |
| wsrep_evs_repl_latency       | 0/0/0/0/0                            |
| wsrep_evs_state              | OPERATIONAL                          |
| wsrep_flow_control_paused    | 0.000000                             |
| wsrep_flow_control_paused_ns | 0                                    |
| wsrep_flow_control_recv      | 0                                    |
| wsrep_flow_control_sent      | 0                                    |
| wsrep_gcomm_uuid             | d1198d28-367a-11e8-a0ac-2382228e259f |
| wsrep_incoming_addresses     | 1.1.1.1:3306,2.2.2.2:3306            |
| wsrep_last_committed         | 0                                    |
| wsrep_local_bf_aborts        | 0                                    |
| wsrep_local_cached_downto    | 18446744073709551615                 |
| wsrep_local_cert_failures    | 0                                    |
| wsrep_local_commits          | 0                                    |
| wsrep_local_index            | 1                                    |
| wsrep_local_recv_queue       | 0                                    |
| wsrep_local_recv_queue_avg   | 0.100000                             |
| wsrep_local_recv_queue_max   | 2                                    |
| wsrep_local_recv_queue_min   | 0                                    |
| wsrep_local_replays          | 0                                    |
| wsrep_local_send_queue       | 0                                    |
| wsrep_local_send_queue_avg   | 0.000000                             |
| wsrep_local_send_queue_max   | 1                                    |
| wsrep_local_send_queue_min   | 0                                    |
| wsrep_local_state            | 4                                    |
| wsrep_local_state_comment    | Synced                               |
| wsrep_local_state_uuid       | 09e3b6c8-343c-11e8-87cf-07a9813fdf95 |
| wsrep_protocol_version       | 8                                    |
| wsrep_provider_name          | Galera                               |
| wsrep_provider_vendor        | Codership Oy <info@codership.com>    |
| wsrep_provider_version       | 25.3.23(r3789)                       |
| wsrep_ready                  | ON                                   |
| wsrep_received               | 10                                   |
| wsrep_received_bytes         | 1081                                 |
| wsrep_repl_data_bytes        | 0                                    |
| wsrep_repl_keys              | 0                                    |
| wsrep_repl_keys_bytes        | 0                                    |
| wsrep_repl_other_bytes       | 0                                    |
| wsrep_replicated             | 0                                    |
| wsrep_replicated_bytes       | 0                                    |
| wsrep_thread_count           | 2                                    |
+------------------------------+--------------------------------------+

node1 my.cnf:

log_bin = /var/mysql/log/mysql-bin.log
max_binlog_size = 100M
expire_logs_days=3
max_binlog_cache_size = 2G
binlog_cache_size = 32K
max_binlog_stmt_cache_size = 2G
binlog_stmt_cache_size = 32K
binlog_format=row
default-storage-engine=InnoDB
innodb_autoinc_lock_mode=2
query_cache_size=0
query_cache_type=0
innodb_flush_log_at_trx_commit=0
wsrep_on=ON
wsrep_slave_threads=1
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="my_cluster"
wsrep_cluster_address="gcomm://2.2.2.2"
# also tried below
#wsrep_cluster_address="gcomm://2.2.2.2,1.1.1.1"
wsrep_sst_receive_address=1.1.1.1:4444
wsrep_provider_options='ist.recv_addr=1.1.1.1:4568;'
wsrep_sst_method=rsync
wsrep_sst_donor="node2,"
wsrep_node_address="1.1.1.1"
wsrep_node_name="node1"

node2 my.cnf:

log_bin = /var/mysql/log/mysql-bin.log
max_binlog_size = 100M
expire_logs_days=3
max_binlog_cache_size = 2G
binlog_cache_size = 32K
max_binlog_stmt_cache_size = 2G
binlog_stmt_cache_size = 32K
server-id = 10
relay-log = /var/mysql/log/mysql-relay-bin.log
replicate-ignore-db = mysql
binlog_format=row
default-storage-engine=InnoDB
innodb_autoinc_lock_mode=2
query_cache_size=0
query_cache_type=0
innodb_flush_log_at_trx_commit=0
wsrep_on=ON
wsrep_slave_threads=1
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="my_cluster"
# for bootstrapping
wsrep_cluster_address="gcomm://"
#wsrep_cluster_address="gcomm://2.2.2.2,1.1.1.1"
wsrep_sst_receive_address=2.2.2.2:4444
wsrep_provider_options='ist.recv_addr=2.2.2.2:4568;'
wsrep_sst_method=rsync
wsrep_node_address="2.2.2.2"
wsrep_node_name="node2"

Best Answer

Your node2 needs to have log-slave-updates=1.

See also SeveralNines' article on the topic which states:

log-slaves-updates=1 is necessary for the writes to propagate from the slave to the other galera nodes