SQLite database replication on the same server

replicationsqlite

We have an application written in C and we are looking for a way to do transparent replication of SQLite database on one server (but on different disks). I read lots of guides about replication across multiple servers and most of what I've found is about RDBMSs such as MySQL database.

This post is the only one I've found about replication specifically on SQLite, where some solutions were suggested. For example, someone suggested using Litereplica which seems to be exactly what I am looking for, but after spending about 2,3 days, it didn't work (or maybe I couldn't use it properly). Another solution was Rqlite, but that is not on the same server.

I am not a SQLite expert and I just started using it, and I really stuck on this. Is there any other solution to replicate the SQLite database on the same server? For example, what we need is e.g., what we write in the main db, will be written in the replicated db/dbs as well. Also, for read, if the main db did not work, it should read from the slave/backup db…

UPDATE_1

I used the ATTACH to copy all the data from the main db to the backup db. Here is what I've done in SQLite command prompt:

ATTACH '/root/litereplica/litereplica-master/sqlite3.6/fuel_transaction_1_backup.db' AS backup;  //This attached a new empty db to the main db

CREATE TABLE backup.fuel_transaction_1 AS SELECT * FROM main.fuel_transaction_1;

Now, Is there any way to sync this two databases? so that e.g., if I delete a row from a table in main db, it deletes the same row from a table in backup db?

Best Answer

Searching for the same topic, I came across a mention in Using SQLite: Small. Fast. Reliable. Choose Any Three.:

Replication

SQLite has no internal support for database replication or redundancy. Simple replication can be achieved by copying the database file, but this must be done when nothing is attempting to modify the database. Replication systems can be built on top of the basic database API, but such systems tend to be somewhat fragile. Overall, if you're looking for real-time replication — especially at a transaction-safe level — you'll need to look at a more complex RDBMS platform

Related Solutions

How to prevent SQLite database locks

That page you linked, besides being quite old, talks about accesses from the same process through the same database connection (or through multipe connections in shared cache mode, which you should not use).

When not in WAL mode, multiple connections can read from the same database, but a writing transaction is exclusive, i.e., no other readers or writers are allowed.

In WAL mode, a writer and readers do not block each other, but there is still only one writer allowed.

Mysql – Master/Master/Slave Replication

Using the topology you mentioned from the MySQL Documentation

sxjn

Let's setup the first scenario

EXAMPLE IPs for Each DB Server

Master1 : 10.20.30.40
Master2 : 10.20.30.50
Slave_1 : 10.20.30.60
replication user is repluser@'%'
replication password is replpass

Make sure binary logging is enabled on all the Slaves

We will do the following

Promote Master2 to Master1
Promote Slave_1 to Master2
Demote Master1 to Slave1

Step 01 : On Master2, run the following

mysql> SET GLOBAL read_only = 1;
mysql> STOP SLAVE;
mysql> RESET SLAVE;
mysql> CHANGE MASTER TO master_host='';
mysql> FLUSH TABLES;
mysql> SET GLOBAL read_only = 0;

Step 02 : Move your CNAME/VIP to Master2

Step 03 : On Slave_1, run mysql> RESET MASTER; FLUSH TABLES;

Step 04 : On Slave_1, dump the data

NEW_MASTER_HOST="10.20.30.60"
MYSQL_USER=root
MYSQL_PASS=rootpassword
MYSQL_CONN="-u${MYSQL_USER} -p${MYSQL_PASS}"
MYSQLDUMP_OPTIONS="--single-transaction"
MYSQLDUMP_OPTIONS="${MYSQLDUMP_OPTIONS} --routines"
MYSQLDUMP_OPTIONS="${MYSQLDUMP_OPTIONS} --triggers"
MYSQLDUMP_OPTIONS="${MYSQLDUMP_OPTIONS} --master-data=1"
MYSQLDUMP_OPTIONS="${MYSQLDUMP_OPTIONS} --all-databases"
echo "STOP SLAVE;" > MySQLData.sql
echo "CHANGE MASTER TO master_host='${NEW_MASTER_IP}'," >> MySQLData.sql
echo "master_port=3306," >> MySQLData.sql
echo "master_user='repluser'," >> MySQLData.sql
echo "master_password='replpass'," >> MySQLData.sql
echo "master_log_file='bogus'," >> MySQLData.sql
echo "master_log_pos=1;" >> MySQLData.sql
mysqldump ${MYSQL_CONN} ${MYSQLDUMP_OPTIONS} >> MySQLData.sql
echo "START SLAVE;" >> MySQLData.sql
gzip MySQLData.sql

Step 05 : When Master1 comes back up, rsync or scp MySQLData.sql.gz from Slave_1 to Master1

Step 06 : Login to MySQL on the Master1 and setup it up to replicate from Slave_1

Don't worry about the real binary log filename and position.

Using --master-data=1 embeds CHANGE MASTER TO command with the real coordinates on line 23 of a standard dump.

You can see it with the following

less MySQLData.sql.gz | head -35 | tail -1

Step 07 : Load the data into Master1

MYSQL_USER=root
MYSQL_PASS=rootpassword
MYSQL_CONN="-u${MYSQL_USER} -p${MYSQL_PASS}"
gzip -d < MySQLData.sql.gz | mysql ${MYSQL_CONN}

Step 08 : Login to MySQL on Master1

mysql> SHOW SLAVE STATUS\G

and Make sure Slave_IO_Running and Slave_SQL_Running both say Yes

EPILOGUE

When Done, the topology should be

Master1 : 10.20.30.50
Master2 : 10.20.30.60
Slave_1 : 10.20.30.40

DISCLAIMER

Please try this out on Test Servers before deploying to Production

GIVE IT A TRY !!!

Note: I would advise that you have multiples slaves

One for Nightly Backups
The others for load balancing SELECTs