I am going to disagree on large and complicated queries with datagod here. I see these only as problems if they are disorganized. Performance-wise, these are almost always better because the planner has much more freedom in how to go about retrieving the information. However, large queries do need to be written with maintainability in mind. In general, I have found that simple, well-structured SQL to be easy to debug even when a single query goes on for 200+ lines. This is because usually you have a pretty good idea of what kind of problem you are dealing with so there are only a few areas in the query that you have to check.
The maintenance problems, IME, come in when the structure of SQL breaks down. Long, complex queries in subselects impairs readability and troubleshooting, as do inline views, and both of these should be avoided in long queries. Instead, use VIEWs if you can (note if you are on MySQL, views do not perform all that well, but on most other db's they do), and use common table expressions where those don't work (MySQL doesn't support these btw).
Long complex queries work pretty well both from a maintainability and performance case where you keep your where clauses simple, and where you do as much as you can with joins instead of subselects. The goal is to make it so that "records aren't showing up" gives you a few very specific places in the query to check (is it getting dropped in a join or filtered out in a where clause?) and so the maintenance team can actually maintain things.
Regarding scalability, keep in mind that the more flexibility the planner has, that's a good thing too....
Edit: You mention this is MySQL, so views are unlikely to perform that well and CTE's are out of the question. Additionally the example given is not particularly long or complex so that's no problem.
As first step I merge the two product tables. I need an full outer join on product1 and product2 tables. To obtain on outer join in mysql we can union a left join and a right join. Start with a left join. In the following query IFNULL(p1.key1, p2.key1) is useful to merge two key column coming from the two product tables and IFNULL(, 'NaN') is useful to obtain 'NaN' in the output.
select IFNULL(p1.key1, p2.key1) as key1,
IFNULL(p1.key2, p2.key2) as key2,
IFNULL(p1.serial, p2.serial) as serial,
IFNULL(p1.product_data, 'NaN') as product_data1,
IFNULL(p2.product_data, 'NaN') as product_data2
from product1 p1
left join product2 p2 on p1.key1 = p2.key1 and p1.key2 = p2.key2 and p1.serial = p2.serial;
To obtain a full outer join I need to repeat the previous query with 'right join' and union the result with the 'left join'
select IFNULL(p1.key1, p2.key1) as key1,
IFNULL(p1.key2, p2.key2) as key2,
IFNULL(p1.serial, p2.serial) as serial,
IFNULL(p1.product_data, 'NaN') as product_data1,
IFNULL(p2.product_data, 'NaN') as product_data2
from product1 p1
left join product2 p2 on p1.key1 = p2.key1 and p1.key2 = p2.key2 and p1.serial = p2.serial
union
select IFNULL(p1.key1, p2.key1) as key1,
IFNULL(p1.key2, p2.key2) as key2,
IFNULL(p1.serial, p2.serial) as serial,
IFNULL(p1.product_data, 'NaN') as product_data1,
IFNULL(p2.product_data, 'NaN') as product_data2
from product1 p1
right join product2 p2 on p1.key1 = p2.key1 and p1.key2 = p2.key2 and p1.serial = p2.serial;
For convenience, I create a view with the previous query:
create or replace view p12 as
select IFNULL(p1.key1, p2.key1) as key1,
IFNULL(p1.key2, p2.key2) as key2,
IFNULL(p1.serial, p2.serial) as serial,
IFNULL(p1.product_data, 'NaN') as product_data1,
IFNULL(p2.product_data, 'NaN') as product_data2
from product1 p1
left join product2 p2 on p1.key1 = p2.key1 and p1.key2 = p2.key2 and p1.serial = p2.serial
union
select IFNULL(p1.key1, p2.key1) as key1,
IFNULL(p1.key2, p2.key2) as key2,
IFNULL(p1.serial, p2.serial) as serial,
IFNULL(p1.product_data, 'NaN') as product_data1,
IFNULL(p2.product_data, 'NaN') as product_data2
from product1 p1
right join product2 p2 on p1.key1 = p2.key1 and p1.key2 = p2.key2 and p1.serial = p2.serial;
To check we can try with:
mysql> select * from p12 where key1 = 1 and key2 =1 ;
+------+------+--------+---------------+---------------+
| key1 | key2 | serial | product_data1 | product_data2 |
+------+------+--------+---------------+---------------+
| 1 | 1 | 0 | 15.556 | 5.556 |
| 1 | 1 | 1 | 14.996 | 4.996 |
| 1 | 1 | 2 | 12.556 | NaN |
| 1 | 1 | 3 | 15.669 | NaN |
+------+------+--------+---------------+---------------+
So the he final query is:
select mi.key1, mi.key2, p12.serial, mi.info1, mi.info2, mi.date, p12.product_data1, p12.product_data2
from main_info mi
inner join p12 on mi.key1 = p12.key1 and mi.key2 = p12.key2
order by mi.key1, mi.key2, p12.serial;
+------+------+--------+-------+-------+----------+---------------+---------------+
| key1 | key2 | serial | info1 | info2 | date | product_data1 | product_data2 |
+------+------+--------+-------+-------+----------+---------------+---------------+
| 1 | 1 | 0 | 15 | 90 | 20120501 | 15.556 | 5.556 |
| 1 | 1 | 1 | 15 | 90 | 20120501 | 14.996 | 4.996 |
| 1 | 1 | 2 | 15 | 90 | 20120501 | 12.556 | NaN |
| 1 | 1 | 3 | 15 | 90 | 20120501 | 15.669 | NaN |
| 1 | 2 | 0 | 14 | 92 | 20120601 | 12.556 | 2.556 |
| 1 | 2 | 1 | 14 | 92 | 20120601 | 13.335 | 3.335 |
| 1 | 2 | 2 | 14 | 92 | 20120601 | NaN | 2.56 |
| 1 | 2 | 3 | 14 | 92 | 20120601 | NaN | 3.556 |
| 1 | 3 | 1 | 15 | 82 | 20120801 | 12.225 | 2.225 |
| 1 | 3 | 2 | 15 | 82 | 20120801 | 13.556 | 3.556 |
| 1 | 3 | 3 | 15 | 82 | 20120801 | 14.556 | NaN |
| 2 | 1 | 0 | 17 | 90 | 20130302 | 12.556 | NaN |
| 2 | 1 | 1 | 17 | 90 | 20130302 | 13.553 | NaN |
| 2 | 1 | 2 | 17 | 90 | 20130302 | 12.335 | NaN |
| 2 | 2 | 0 | 16 | 88 | 20130601 | NaN | 2.556 |
| 2 | 2 | 1 | 16 | 88 | 20130601 | NaN | 3.553 |
+------+------+--------+-------+-------+----------+---------------+---------------+
Best Answer
I have been in a similar situation and highly recommend looking into MongoDB or HBase for your needs. The others (Cassandra/Redis) are good too, but HBase and MongoDB have been battle-tested in large environments, have a large community, good documentation (MongoDB's is better) and generally fulfill the three requirements you are looking for.