MySQL UTF-8 Encoding Problem

MySQL should include a number of starting point configuration files for my.cnf that can adjust the memory usage. The files are named my-small.cnf, my-medium.cnf, my-large.cnf, and my-huge.cnf.

There is an online MySQL memory usage calculator at http://www.mysqlcalculator.com/. By comparing the starting point files and using the memory usage calculator you should be able to adjust your my.cnf configuration file to accomodate your server's memory constraints.

For more information about what each parameter does, you can reference the MySQL documentation available online. Here is a link to the MySQL documentation for MySQL server 5.1: http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html

You need to take the data from UTF-8 and convert it into UCS-2LE using something like iconv. For example, using the character in your example:

echo "010000: dcb3" | xxd -r -s -0x10000 | iconv -f "UTF-8" -t "UCS-2LE" | xxd
0000000: 3307 

Now I'm not sure what character UTF-8 \xdcb3 is, but apparently it's correct translation to UCS-2LE is \U0733. If you have \0xDCB3 in the SQL Server it means it was not translated into UCS-2LE before import. You should not have surrogates in the NVARCHAR fields, UCS-2 is "surrogate agnostic". See UCS-2 vs. UTF-16 (not quite Kramer vs. Kramer).

I'm not an expert in the MySQL tool set so I can't say what step is missing that was supposed to do the iconv.

Update

to locate the records with surrogates you must turn to the binary representation, since any character function will threat the surrogates as 'special':. Luckily the string manipulation functions work on binary too with the expected semantics. Eg. CHARINDEX:

insert into test(a) values  (N'a');
insert into test(a) values  (NCHAR(0xdc83));
insert into test(a) values  (N'b');
go

select * from test where charindex(0x83dc, cast(a as varbinary(8000))) > 0;