I have this query:
select m.id meal_id,
SUM(FORMAT( ((((fi.calories / fi.weight_in_grams) * w.GRAMS) * i.amount) / r.serves) * mr.servings , 3)) calories
from meals m
join meal_ethnic_types met
on met.meal_id = m.id
join meal_protein_types mpt
on mpt.meal_id = m.id
join meal_recipes mr
on mr.meal_id = m.id
join recipes r
on r.id = mr.recipe_id
join ingredients i
on i.recipe_id = r.id
join food_items fi
on fi.id = i.food_item_id
join WEIGHT w
on w.NDB_NUM = fi.usda_code and w.MEASURE_CODE = i.measurement_id
join meal_meal_types mmt
on m.id = mmt.meal_id
where m.is_active = 1
and (m.company_id = 44 or m.company_id is null)
and (m.location_id = 167 or m.location_id is null)
and (m.member_id = 2698 or m.member_id is null)
and (m.user_id = 3151 or m.user_id is null)
and mmt.meal_type_id = 2
and calories <= 343
and m.cpf_id = 2
and (mpt.protein_type_id in (2, 4))
and (met.ethnic_type_id in (1, 7, 5, 4, 9, 6))
group by m.id
order by rand()
limit 0, 50;
So my issue is that on my development box, and on the RDS staging database the system query uses an index on the WEIGHT join. The explain returns this:
1 SIMPLE m ref PRIMARY,cpf_id,company_id,location_id,member_id,user_id,is_active cpf_id 4 const 273 Using where; Using temporary; Using filesort
1 SIMPLE mpt ref meal_id,protein_type_id meal_id 4 staging_myplan.m.id 1 Using index condition; Using where
1 SIMPLE mmt ref meal_id,meal_type_id meal_id 4 staging_myplan.m.id 1 Using index condition; Using where
1 SIMPLE mr ref meal_id,recipe_id meal_id 4 staging_myplan.m.id 1 Using index condition
1 SIMPLE r eq_ref PRIMARY PRIMARY 4 staging_myplan.mr.recipe_id 1 Using where
1 SIMPLE met ref meal_id,ethnic_type_id meal_id 4 staging_myplan.m.id 1 Using index condition; Using where
1 SIMPLE i ref food_item_id,recipe_id recipe_id 4 staging_myplan.r.id 2 Using index condition
1 SIMPLE fi eq_ref PRIMARY,usda_code PRIMARY 8 staging_myplan.i.food_item_id 1 Using where
1 SIMPLE w ref ndbnum,MEASURE_CODE ndbnum 12 staging_myplan.fi.usda_code 1 Using index condition; Using where
Yet the same exact query on production RDS (both a instance install and on an RDS) Takes forever and returns this explain:
1 SIMPLE m ref PRIMARY,cpf_id,company_id,location_id,member_id,user_id,is_active cpf_id 4 const 273 Using where; Using temporary; Using filesort
1 SIMPLE mpt ref meal_id,protein_type_id meal_id 4 don_moreactive.m.id 1 Using where
1 SIMPLE mmt ref meal_id,meal_type_id meal_id 4 don_moreactive.m.id 1 Using where
1 SIMPLE met ref meal_id,ethnic_type_id meal_id 4 don_moreactive.m.id 1 Using where
1 SIMPLE mr ref meal_id,recipe_id meal_id 4 don_moreactive.m.id 1 Using where
1 SIMPLE r eq_ref PRIMARY PRIMARY 4 don_moreactive.mr.recipe_id 1 Using where
1 SIMPLE i ref food_item_id,recipe_id recipe_id 4 don_moreactive.r.id 2 Using where
1 SIMPLE fi eq_ref PRIMARY,usda_code PRIMARY 8 don_moreactive.i.food_item_id 1 Using where
1 SIMPLE w ALL MEASURE_CODE NULL NULL NULL 13162 Using where; Using join buffer
So as you can see the production databases are not using the index and are attempting to use where and the join buffer.
I've compared the keys and indexes on all database servers and they're identical.
Any clues?
Staging Table:
CREATE TABLE `WEIGHT` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`NDB_NUM` varchar(10) NOT NULL,
`MEASURE_CODE` varchar(10) NOT NULL DEFAULT '',
`GRAMS` decimal(11,2) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `ndbnum` (`NDB_NUM`) USING BTREE,
KEY `MEASURE_CODE` (`MEASURE_CODE`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=12920 DEFAULT CHARSET=latin1 COMMENT='InnoDB free: 1167360 kB; (`NDB_NUM`) REFER `FVG_NEW_NUTRITIO'
WEIGHT 0 PRIMARY 1 id A 12774 NULL NULL BTREE
WEIGHT 1 ndbnum 1 NDB_NUM A 12774 NULL NULL BTREE
WEIGHT 1 MEASURE_CODE 1 MEASURE_CODE A 3193 NULL NULL BTREE
Production Table:
CREATE TABLE `WEIGHT` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`NDB_NUM` varchar(10) NOT NULL,
`MEASURE_CODE` varchar(10) CHARACTER SET utf8 NOT NULL,
`GRAMS` decimal(11,2) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `ndbnum` (`NDB_NUM`) USING BTREE,
KEY `MEASURE_CODE` (`MEASURE_CODE`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=12920 DEFAULT CHARSET=latin1 COMMENT='InnoDB free: 1167360 kB; (`NDB_NUM`) REFER `FVG_NEW_NUTRITIO'
WEIGHT 0 PRIMARY 1 id A 13076 NULL NULL BTREE
WEIGHT 1 ndbnum 1 NDB_NUM A 13076 NULL NULL BTREE
WEIGHT 1 MEASURE_CODE 1 MEASURE_CODE A 3269 NULL NULL BTREE
STAGING:
index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,subquery_materialization_cost_based=on,use_index_extensions=on
PRODUCTION:
index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on
Best Answer
The difference is the
CHARACTER SET
of theMEASURE_CODE
column.In staging it is
latin1
, in production it isutf8
.If character set of columns in the join (
w.MEASURE_CODE = i.measurement_id
) doesn't match, the server converts one of the columns to a different character set. This is done row-by-row and prevents using the index.Here is a simple set up to demonstrate the problem.
When character set matches:
When character set doesn't match:
You can see the difference in
possible_keys
andref
columns. Note, thatref
is actual column in the first variant (db.Test1.Code1
) andfunc
in the second variant. I believe thisfunc
is implicit conversion fromlatin1
toutf8
.This is tested on: