Mysql – Optimize schema and query for overlapping date ranges query

MySQLperformancequery-performance

We have a database where we track the price of products from different sellers. If I strip everythin else it looks like this:

I need to make a report where given a date range I have to return a row for every day and the prices for every seller. There will be days that some seller don't have a price for a product and there will be days that a seller will have more than one price for a given product.

I'm doing it by creating a temp table with the date ranges then join them with the sellers and prices for this but once product_price have a couple thousands rows it get VERY slow (about 4'' to return a 200 rows) and I don't know how to optimize it to get it fast enought.

I can't think a workaround for the temp table nor which index should I create.

schema and query here: DB-fiddle

Best Answer

You have only primary keys for tables and indexes related to the FK restrictions but no indexes related to the queries you run.

SELECT qtm.date_i
     , s.name
     , pp.* 
  FROM query_time_window   AS qtm
  JOIN seller              AS s   -- some ON condition is strictly recommended
  LEFT JOIN product_price  AS pp  ON pp.seller_id = s.id -- most conditions are moved to WHERE
 WHERE ( pp.product_id = 1 OR pp.product_id IS NULL )
   AND ( pp.created_at <= qtm.date_f   -- I've changed '!a>b' to 'a<=b'
         AND ( pp.cancelled_at IS NULL 
               OR pp.cancelled_at >= qtm.date_i) -- I've changed '!a<b' to 'a>=b'
       )
 ORDER BY qtm.date_i, s.id;

Table query_time_window should have an index (date_i, date_f) or (date_f, date_i)
Table product_price should have an additional index (product_id, seller_id, created_at, cancelled_at) - an exact order of the columns depend on the real data. You can try different combinations and choose the fastest one.

Related Solutions

Mysql – optimize big sql query

All time we use lot of SQL queries. But we don’t consider about its performance. If we follow some tips then our query will be more efficient. Some of them are.

Use views and stored procedures instead of heavy-duty queries.
Try to use constraints instead of triggers, whenever possible.
Use table variables instead of temporary tables.
Try to use UNION ALL statement instead of UNION

For more http://cybarlab.blogspot.com/2013/02/sql-queries-optimization-tips.html

Thanks n regard

MySQL query to evaluate ranges in date and time elements of a datetime column separately

This actually turned out to be fairly painful. :) The details of this logic ran though my head most of last night... but in my current sleep-deprived state, I think I have devised a solution where every case is handled, and which does what we need in a reasonably-optimal fashion.

It returns the correct values for all of the test cases.

You need an index on (EventTime) so that the optimizer can turn the date-level tests into constants and use that index to find the matching rows by date range... then the server can filter the date-matched rows based on their time match in the CASE statement of the where clause.

In the WHERE clause, the CASE expression matches a condition in our data, and will yield a boolean value based on the truthiness of the expressions in the THEN. This value will include a row if TRUE and exclude the row if FALSE.

I've used variables, but that's mostly for clarity... assuming there's any clarity to be found in what I've written, below, and to prevent errors since the literals are needed in so many different places.

SET @TargetDateTime = '2013-09-26 00:00:00';
SET @DateRange = 7;
SET @TimeRange = 15;

SELECT E.*
 FROM EventTable E
      -- must be at or after midnight the morning of the first valid day
WHERE EventTime >= DATE_SUB(DATE(@TargetDateTime), INTERVAL @DateRange DAY)
      -- must be before midnight on the morning after the first valid day 
  AND EventTime <  DATE_ADD(DATE(@TargetDateTime), INTERVAL (@DateRange + 1) DAY)
      -- now, hold my beer and watch this...
  AND CASE 
  -- easy first case, the time portions of both values are equal
  WHEN TIME(EventTime) = TIME(@TargetDateTime) 
  -- in each case if the expression in the "THEN" evaluates to something true, 
  -- then the row will be returned if the other conditions in WHERE are also met
  THEN TRUE
  -- next case, if the event appears to be earlier than the target, 
  -- we match the row IF either...
  WHEN TIME(EventTime) < TIME(@TargetDateTime)
  -- ...the simple case, difference in seconds is within range
  THEN TIME_TO_SEC(@TargetDateTime) - TIME_TO_SEC(EventTime)
       BETWEEN 0 AND (@TimeRange * 60)
       OR 
  -- ...the complicated case, when we span midnight 
       86400 + TIME_TO_SEC(EventTime) - TIME_TO_SEC(@TargetDateTime) <= (@TimeRange * 60)
  -- next case, event appears later than target, we match if either...
  WHEN TIME(EventTime) > TIME(@TargetDateTime)
  -- ... simple subtraction returns a value in range
  THEN TIME_TO_SEC(EventTime) - TIME_TO_SEC(@TargetDateTime)
       BETWEEN 0 AND (@TimeRange * 60)
       OR
  -- ... we wrap around midnight
      86400 + TIME_TO_SEC(@TargetDateTime) - TIME_TO_SEC(EventTime) <= (@TimeRange * 60)
  -- ELSE isn't encountered unless we run into null values, in which case, no match
  ELSE FALSE END; /* END of CASE expression */

A working demo of the final version is here: http://sqlfiddle.com/#!2/7b8e5/2

It is usually bad form in a WHERE clause to use columns as arguments to functions, because this prevents the optimizer from using an available index, since it has to evaluate the data in every row of the table in order to evaluate the function. There's no harm, here, though, since an index can't help us match the times. This is because DATETIME values are on the left hand side of the field, as stored, and the time values are on the right, and an index stops helping as soon as we encounter irrelevant information from left to right. (That's usually discussed in the context of multi-column indexes, but it also holds true, here.)

It's also often bad form in a WHERE clause to use a CASE expression because that too may prevent the optimizer from choosing the optimal path because the CASE can prevent the optimizer from forming a proper understanding of the logic, but this is acceptable here because we have nothing but AND in the WHERE clause and two other conditions that are easily satisfied by an index.

Update: This code and the answer went through several iterations before arriving at its current (working) form, as you can see in the comments.

Essentially, what we are doing is considering the three cases than can be identified when trying to compare wall-clock time of day: the times are equal, A is earlier than B, or B is earlier than A. It seems like "absolute value" might come into play, but the problem we encounter is that when a time value has no date component, earlier and later aren't meaningful concepts when the times might span midnight. Without a date component, 00:05:00 and 23:55:00 are 23 hours and 50 minutes apart. With a date component, they could also be "n days and 10 minutes" apart. When the date component is missing but uninteresting (or already handled, in this case), we first check the simple case, subtracting the "earlier" from the "later" to see if they're within range, but when they're not, we add 86400 seconds to the "earlier" component, essentially making it "the same time, the following day"... then the subtraction can be done to see if the times are close enough together when they're calculated as having spanned midnight.

Best Answer

Related Solutions

Mysql – optimize big sql query

MySQL query to evaluate ranges in date and time elements of a datetime column separately

Related Question