Ubuntu – Check real size of USB thumb drive

hardwarehardware-testmicro-sdsd cardusb-drive

I recently read a lot about fake MicroSD card and USB thumb drives that claim to have a lot of space (even if you ask your computer), while physically offering way less. I recently bought a SanDisk USB drive (128 GB claimed) and want to test its size. It's not bought via ebay or something, but I really want to test the real size before using it productively.

I could just copy stuff on it, copy it back and look if the files are okay. I could also automate it with Hashes and stuff. But I hoped there is a more accurate solution. I read that for Windows, H2testw does the trick. Is there an easy way to test this on Ubuntu/Linux? A specialized, well working tool maybe?

Update: Just to be clear, the idea is to verify that the size the linux system gets told by the controller is correct (so no data will be lost). It's not like I want to see if I get 128 GB instead of 127.3 GB. I want to test if all data I write will be readable again. Unfortunately I can only find a few information about this on English tech sites. There are good German sources, though. I'm actually searching for an application like those, but for Ubuntu/Linux: https://www.raymond.cc/blog/test-and-detect-fake-or-counterfeit-usb-flash-drives-bought-from-ebay-with-h2testw/

Update2:
I tried to get together some sources in English. I did not read all of them in detail, due to missing time.

Update3: Explanations

Due to the strange critics below, some explanations.

What is the problem and why does dd alone not solve it?

This is a reaction to

"Clearly figure out what is the problem you're trying to solve and what is the definition of "fake drive"."

It seems that some people do not understand the problem. So I try to explain it as short as I can in details, though I think this is much to the extend of my question.

The capacity of usb devices your operating system or unix tools give you, can be wrong. This is fatal, since your OS regulates how much data you can send it to. Send more data than it can really hold, you'll get a data loss. This is a problem. So, why can this happen?

You do not need to know the USB-Protocol well to unterstand the problem. Serial Interfaces have the common property, that the client device (the usb drive) will need to tell its own capacity via this serial interface. This means that the client device needs it's own controller with some knowledge about the devices purpose and, in this case, it's capacity. It also decides what is done, when it receives the command to store something. If the controller is programmed that way, it can just ignore the command or overwrite something with the data.

What does this mean? Whatever your unix tools tell you about the capacity of the drive: It's what the tools asked the drive, nothing more. This is what h2testw was invented for: It tests the real size with a method explained later on, and compares it to what the drive says. If this is not the same, you may have a data loss, because all your common operations to store data, rely on the information of your operating system, which just asks the controller. Why just ask? Testing needs time and overwrites all data on the drive. So it's natural that an Operating System needs to rely on this information.

To check the real capacity like h2testw, you indeed can use dd to write data on the drive, read it again, and see if it's the same you wrote. Totally legit. The nature of hardware and the drive make it more complicated. Consider write-caches for example. You need to ensure that you do not read from the cache. This is just one example of why it is not as easy as it looks. Also think that just writing zeros means a low entropy of information, which can be reconstructed when reading. It's just not that easy in detail. You can still do it manually, of course.

But why, when you can automate things? Why to the work? f3 as proposed in my answer below, implements tons of thoughts of many contributors (consider that it kind of extended h2testw) and it also implements several methods with different trade-offs. The developer figured out the tricks of different fake drives (aka counterfeit drives) they had at hand. So, while I understand the theory and the problem (seemingly since the problems are well explained in german tech media, but not in english speaking media), I do not pretend to understand everything, which is why I mentioned it above. It's just the theory I understand, and I'm more of a software guy. But as a student of informatics I understand it well enough to see the problem.

"Try to understand basic Unix utilities"

Actually I answered this one already, but to make it clear: Unix tools just use the USB-Protocol (for USB-devices only, of course) to gather information. It does not make sense to do more than that.

Does it help to only buy from trustes suppliers?

tl;dr: It does not.

"When it comes to buying goods, just like it comes to any form of security, consider finding a trusted seller and buy drives only from them."

Security (and safety) is NOT about trust! It's about verification and validation! Sorry but this is so wrong in so many ways.

Assume you buy via a trusted seller. A few questions:

Did the supplier test the hardware to ensure there is no data loss? Does re recognize when he buys fake drives and sells them? Not necessarily.
Is it possible that he buys stuff he doesn't know is fake? Totally, look at the recent ryzen fakes: https://www.pcgamer.com/beware-of-fake-ryzen-processors-selling-on-amazon/ , https://www.heise.de/newsticker/meldung/Direkt-von-Amazon-Faelschungen-von-AMDs-Ryzen-Prozessoren-im-Umlauf-3772757.html
If I loose my presentation in the drive and screw up the presentation, will my trusted supplier go back in time and rescue me? It will probably replace the drive, since the last time-travelling DeLorean was destroyed in 1885.

Other stuff

"This question really seems to be more like "promo" for what OP likes, and seems that OP is much less interested in actually testing the drives."

This is ridiculous. I was searching specifically for a similar tool to h2testw that also runs on linux. And yes, that's what I'd "like", helpful answer, so sorry. I had no idea that the english speaking press is not that aware of such issues and was lucky to find something like that later on. This is not a promo, but actually it seems you could use one.

Best Answer

f3 - Fight Flash Fraud

There is only one alternative I found, but I think this is even a better one than the original h2testw tool for MS Windows. Fortunately, it is really easy to use, even from command line. There are GUIs available, though. There is also a lot of information about the implementation and the problem with fake drives on the tools website.

f3 offer two methods:

f3probe method: Much faster
h2testw method: Slower. Also test R/W performance. Probably more reliable.

The f3probe method (recomended)

f3probe is one way to test the drives, not as accurate but faster since it does not write on the whole drive. You can read more about it on the tools website. If you want to be 100% sure, better use the h2testw method. As the developer describes on the website:

f3probe is the fastest way to identify fake drives and their real sizes.

And:

Finally, thanks to f3probe being free software, and once f3probe is battle proven, f3probe could be embedded on smartphones, cameras, MP3 players, and other devices to stop once and for all the proliferation of fake flash.

There is also a usage example on the website:

Warning: This will destroy any previously stored data on your disk!

$ sudo f3probe --destructive --time-ops /dev/sdb
[sudo] password for michel: 
F3 probe 6.0
Copyright (C) 2010 Digirati Internet LTDA.
This is free software; see the source for copying conditions.

WARNING: Probing may **demolish data,** so it is more suitable for flash drives out of the box, without files being stored yet. The process normally takes from a few seconds to 15 minutes, but
         it can take longer. Please be patient. 

Bad news: The device `/dev/sdb' is a counterfeit of type limbo

You can "fix" this device using the following command:
f3fix --last-sec=16477878 /dev/sdb

Device geometry:
             *Usable* size: 7.86 GB (16477879 blocks)
            Announced size: 15.33 GB (32155648 blocks)
                    Module: 16.00 GB (2^34 Bytes)
    Approximate cache size: 0.00 Byte (0 blocks), need-reset=yes
       Physical block size: 512.00 Byte (2^9 Bytes)

Probe time: 1'13"
 Operation: total time / count = avg time
      Read: 472.1ms / 4198 = 112us
     Write: 55.48s / 2158 = 25.7ms
     Reset: 17.88s / 14 = 1.27s

Note that it also returns a command that enables you to use the drive with it's real size, using f3fix.

The f3fix tool

f3fix creates a partition that fits the actual size of the fake drive. Use f3probe’s output to determine the parameters for i3fix
sudo f3fix --last-sec=16477878 /dev/sdb

The h2testw method / Testing performance with f3read/f3write

F3 is a collection of tools that deal with fake flash drives. Two of them together implement the h2testw-Method:

f3write [--start-at=NUM] [--end-at=NUM] <PATH>
f3read  [--start-at=NUM] [--end-at=NUM] <PATH>

f3write will ask for the devices claimed size and fill it with generated files with a size of 1gb each. f3read will read all those files and see of they are complete and not broken. As an example the commands I used to test my ~128gb thumb drive:

$ f3write /media/username/1EB8021AB801F0D7/
Free space: 117.94 GB
Creating file 1.h2w ... OK!                           
...
Creating file 118.h2w ... OK!                         
Free space: 0.00 Byte
Average writing speed: 11.67 MB/s

Now to test whether the files are correctly stored:

$ f3read /media/username/1EB8021AB801F0D7/
                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097152/        0/      0/      0
...
Validating file 118.h2w ... 1979488/        0/      0/      0

  Data OK: 117.94 GB (247346272 sectors)
Data LOST: 0.00 Byte (0 sectors)
           Corrupted: 0.00 Byte (0 sectors)
    Slightly changed: 0.00 Byte (0 sectors)
         Overwritten: 0.00 Byte (0 sectors)
Average reading speed: 32.38 MB/s

The test for a drive of this size took about three hours with this method and sometimes caused a heavy disk load on my computer, but it's said to me the most accurate.

Install in Ubuntu

On terminal:

sudo apt install f3

This will bring you: f3brew, f3fix, f3probe, f3read, f3write with their man pages.

This tools are part of the f3 package, which is at least available on Ubuntu 15.10. According to the website, there are some more tools that are available. To get them take a look at the website.
The package comes with short but useful manpages, though I think they miss some information from the website about the difference of f3read/write and f3probe for example, which is why this answer got a little longer.