Ubuntu – Running python portably from usb drive

Where USB flash drives fail is in random writes to the disk. They can be hundreds of times slower than the same operation on traditional hard drives. Note that this used to be the case with SSDs too until around 2008 or 2009 when they gained new controller technologies which make them much faster than tradtional hard drives. These technologies pretty much don't exist on USB flash drives.

Operations that will be particularly slow include:

• Installing new packages, or doing system updates. Expect these to be tens of times slower than traditional hard drives, maybe even 1000s of times slower than SSDs. Upgrading between Ubuntu versions could take several hours or more.
• General browser use, given that your browser caches files to disk all the time.
• Operations which involve copying or moving/altering hundreds of small files.

This article on phoronix compares the performance of various file systems on USB flash drives. Unfortunately, its conclusion is that it doesn't make all that much difference. Theoretically, a log-structured file system should help, but these aren't mainstream enough for me to recommend them for the uninitiated.

One idea is to use a read-only or hybrid file-system instead, which is exactly what you get if you copy the Ubuntu Live CD image (or use UNetBootin to set up a live USB) to your USB flash drive. Look into setting up a Live USB from the Live CD image with persistence (try to find a recent guide to doing so). Of course, you then won't be able to upgrade to a new version of Ubuntu, but that's not necessarily terrible.

Barring any such measures, and keeping a traditional partition type/structure, your main strategy will be to try and minimise writes to the disk at all cost. Ideas for this include:

• Disable journalling on ext4 or set your partitions to "writeback" mode. The drawback to this is that it increases the chances and severity of data loss or corruption in the case of accidental power loss, crash or simply unplugging the drive while it's being used.
• Disable swap, though on a system with sufficient RAM this actually will have much less of an effect than you expect, and on a system with insufficient RAM may cause stability problems.
• Try to minimise on-disk caching by various programs, such as your browser. Reducing the size of the disk cache is not necessarily enough: turning it off all together and using a memory cache (if possible) is much better. If you have lots of RAM, you could try using a tmpfs mount (like a disk partition in RAM) for various tmp and cache directories.

Late getting around to writing this one up.

Preamble

The problem appears to be an inconsistent partition table, often found on factory formatted USB thumb drives. Something about the partition table makes it unsuitable to make a liveUSB.

The Problem

My research into this seems to suggest the problem stems from a partition table written in a DOS 6.x format, a known buggy implementation. Here's explanation from the Linux cfdisk manual page (cfdisk is a text based utility for manipulating disk partition tables):

DOS 6.x WARNING

   The DOS 6.x FORMAT command looks for some information in the first sec‐
   tor of the data area of the partition, and treats this  information  as
   more  reliable than the information in the partition table.  DOS FORMAT
   expects DOS FDISK to clear the first 512 bytes of the data  area  of  a
   partition  whenever a size change occurs.  DOS FORMAT will look at this
   extra information even if the /U flag is given -- we  consider  this  a
   bug in DOS FORMAT and DOS FDISK.

Manifestations of this problem will often include some or all of the following symptoms:

• Testdisk and similar programs will identify the partition table as being made by DOS 5.x
• The boot disk creator in Ubuntu will often refuse to use the USB thumbdrive because it can't find a valid partition table the error message does not state this clearly however.
• Disk utility and Gparted can not delete the partition table because they don't recognise that there is one. Conversely they can't make a partion table either, because they also recognise that there is something there.
• Running fdisk -l on the thumbdrive will often declare that it has no partition table, even though you know there is one and can write data to it if it is still mounting OK.
• Attempts to use the drive and put an ext type partition table on it if successful can sometimes result in inconsistencies - the drive can fail to work on other machines, or suddenly fail to mount on the machine it was formatted on. These appears to be 'dead' USB thumbdrives as the device is firmware is recognised but it fails to mount and the partition table can not be read at all. I suspect a few fairly new thumbdrives have been thrown away, including one of mine before I figured out what the problem was.

Affected devices

Devices that I know from personal experience that have this problem:

Verbatim white 'Stor N Go' 4 and 8 GB devices Many Kingston 4Gb devices Woolworths "essentials" 4GB devices

Common factors with all of these devices is often that they claim Linux OS compatibility, and that they often include some Windows backup software for synchonising Windows data folders.

I don't know why USB thumbdrive manufacturers are supplying devices with what appears to be a partition table format that had problems when it was released. Any later format would probably work without any problems whatsoever, even though it might not be perfect by todays standards.

The Solution

dd if=/dev/zero of=/dev/sdX bs=512 count=1

This little line of code simply writes zeros over the top of the partition table on the device /dev/sdX

You will need to change sdX to the correct device for your faulty USB pendrive.

Warning: Using the incorrect device name will completely delete the partition table from whatever device you point it at - hard drive, external backup drive, anything. This will result in the deletion of any and all partitions that might be on the drive, and cause you a great deal of grief if you don't have all your important data backed up. You Have Been Warned

Once the partition table has been erased you will have to use a program like gparted to create a new one, and to add 1 or more partitions so that the device can be used. In my experience this fix is fast, simple, and permanent. I usually do it on all new USB pendrive devices that I purchase.

It's also possible use Linux command line utilities such as fdisk and extfs to create a new partition table. I shall add some details on this as time allows.

References

• Wikipedia

• cfdisk manpage