BLOBs storage in a dedicated table

Just do

SELECT ItemId
FROM   MyTable
WHERE  Value = 0xAAFF 

You are converting the varchar representation to varbinary(max) which is not correct.

SELECT CONVERT(VARBINARY(MAX), '0xAAFF') returns 0x307841414646 for me for example. It will give you a result based on the character codes in that string representation in your default collation's code page.

You can get the size in bytes of the BLOB column via DBMS_LOB.GETLENGTH(...).

To get a total size in bytes of column for the entire table SUM it up like this:

SELECT SUM(DBMS_LOB.GETLENGTH(my_blob_column)) as total_bytes
FROM my_table

If you've marked the BLOB column as deduplicated then you can run something like the following to get stats on the actual size in bytes used. Compare the used_bytes value before/after your enable settings that could change the size of the BLOB column (eg. dedupe, compression, etc).

declare  
    p_schema_name           varchar(256) DEFAULT 'THE_SCHEMA_NAME';
    p_table_name            varchar(256) DEFAULT 'THE_TABLE_NAME';
    --
    l_segment_name          varchar2(30);
    l_segment_size_blocks   number; 
    l_segment_size_bytes    number; 
    l_used_blocks           number;  
    l_used_bytes            number;  
    l_expired_blocks        number;  
    l_expired_bytes         number;  
    l_unexpired_blocks      number;  
    l_unexpired_bytes       number;  

begin
    FOR rec IN (select segment_name 
                from dba_lobs 
                where owner = p_schema_name
                  and table_name = p_table_name
                order by 1)
    LOOP
        dbms_output.put_line('Segment Name=' || rec.segment_name);

        dbms_space.space_usage( 
            segment_owner           => p_schema_name,  
            segment_name            => rec.segment_name, 
            segment_type            => 'LOB', 
            partition_name          => NULL, 
            segment_size_blocks     => l_segment_size_blocks, 
            segment_size_bytes      => l_segment_size_bytes, 
            used_blocks             => l_used_blocks, 
            used_bytes              => l_used_bytes, 
            expired_blocks          => l_expired_blocks, 
            expired_bytes           => l_expired_bytes, 
            unexpired_blocks        => l_unexpired_blocks, 
            unexpired_bytes         => l_unexpired_bytes 
        );   


        dbms_output.put_line('  segment_size_blocks       => '||  l_segment_size_blocks);
        dbms_output.put_line('  segment_size_bytes        => '||  l_segment_size_bytes);
        dbms_output.put_line('  used_blocks               => '||  l_used_blocks);
        dbms_output.put_line('  used_bytes                => '||  l_used_bytes);
        dbms_output.put_line('  expired_blocks            => '||  l_expired_blocks);
        dbms_output.put_line('  expired_bytes             => '||  l_expired_bytes);
        dbms_output.put_line('  unexpired_blocks          => '||  l_unexpired_blocks);
        dbms_output.put_line('  unexpired_bytes           => '||  l_unexpired_bytes);
    END LOOP;
end;
/

This code is a slightly modified version of the code in Oracle's post explaining how deduplication works. The original is available here: http://www.oracle.com/technetwork/articles/sql/11g-securefiles-084075.html.