The basic problem you have is that everything is recursive and you are attempting to lock pages of data in InnoDB, creating lots of MVCC info. In some case, you may have attempted to lock the same InnoDB page repeatedly. That explains the deadlocks. I have commented on this situation deeply with someone in three separate posts on this same issue:
You need a workaround. You may want to try the following algorithm:
For this example
Step 01) Make a table to collect all father_ids to be updated
DROP TABLE IF EXISTS fathers_to_update;
CREATE TABLE fathers_to_update
(
father_id INT NOT NULL,
PRIMARY KEY (father_id)
);
Step 02) Making a table that will be a FIFO queue. Put the value of 12 into the queue:
DROP TABLE IF EXISTS fathers_to_queue;
CREATE TABLE fathers_to_queue
(
father_id INT,
id INT NOT NULL AUTO_INCREMENT,
PRIMARY KEY (id)
);
INSERT INTO fathers_to_queue (father_id) VALUES (12);
Step 03) Get the Count of fathers_to_queue
SELECT COUNT(1) FROM fathers_to_queue;
If the count is 0, goto Step 07)
Step 04) Remove the front number in fathers_to_queue
SELECT MIN(id),father_id INTO @frontndx,@dad FROM fathers_to_queue;
DELETE FROM fathers_to_queue WHERE id = @frontndx;
Step 05) Collect all father_ids from 161_ft_individual whose id is @dad into the queue
INSERT INTO fathers_to_update SELECT father_id FROM 161_ft_individual WHERE id = @dad;
INSERT INTO fathers_to_queue (father_id) SELECT father_id FROM 161_ft_individual WHERE id = @dad;
Step 06) Go back to Step 03
Step 07) Perform a single UPDATE of all records in 161_ft_individual you collected in fathers_to_update
UPDATE fathers_to_update A
LEFT JOIN 161_ft_individual B
USING (father_id)
SET B.total_leg=B.total_leg+@amount;
That's it. I basically performed a preoder tree traversal back up the hierarchy to all records until father_id was NULL, which makes Step 03 stop collecting father_ids.
Give it a Try !!!
UPDATE 2011-12-16 12:18 EDT
Back on October 24, 2011, I actually wrote an algorithm in the MySQL Stored Procedure language to perform preorder tree traversal: Find highest level of a hierarchical field: with vs without CTEs . Hope it helps !!!
I don't see a lot of opportunity for improvement.
The index you added was probably a big help, because it's being used for the range matching on the WHERE clause (type => range, key => tran_date), and it's being used as a covering index (extra => using index), avoiding the need to seek into the table to fetch the row data.
But since you're using functions to construct the financial_year value for the group by, both the "using filesort" and "using temporary" can't be avoided. But, those aren't the real problem. The real problem is that you're evaluating MONTH(tran_date) 346,485 times and YEAR(tran_date) at least that many times... ~700,000 function calls in one second doesn't seem too bad.
Plan B: I am definitely not a fan of storing redundant data, and I'm dead-set against making the application responsible for maintaining it... but one option I might be tempted to try would be to create a dashboard_stats_by_financial_year table, and use after-insert/update/delete triggers on the transactions1 table to manage keeping those stats current.
That option has a cost, of course -- adding to the amount of time it takes to update/insert/delete a transaction... but, waiting > 1200 milliseconds for stats for your dashboard is a cost, too. So it may come down to whether you want to pay for it now or pay for it later.
Best Answer
I don't think you are going to get anything better than the composite index. However, the key distribution for the composite index could hamper query performance depending on the (hasPhoto,userStatus) combination you choose. Here is how:
Run this Query
This will show you how sparse and how dense each composite index combination is.
Sparse combinations (low rowCount) should use an index scan.
Dense combinations (high rowCount) should use a full table if the Query Optimizer believes it must read too much of the composite index.
You can try this out by running
This will reveal whether the EXPLAIN plan favors an index scan (Extra column
Using where; Using index
), or a full index (or table) scan (select_type of ALL or SIMPLE).