Ubuntu – How to tune Ubuntu for running from a flash drive

cachelinuxUbuntu

I feel like this is a bit of a duplicate of this other question about making caching more aggressive, but I tried a lot of the suggestions there and I just can't figure out why I'm having trouble.

My scenario is that I've got Ubuntu 12.10 on a little USB flash drive I keep on my keychain. (I use it for times when I need to rescue someone's PC or when I want to have my own desktop with my own settings and bookmarks, etc. no matter what PC I find myself in front of).

The problem is that the system is horribly slow.

Now, the USB flash drive is fairly slow, so I know I have to make some concessions. However, I'm currently trying to run it on an i7 machine with 8GB of RAM. "free" and "top" both report that the system is using less than 2.7GB…

          total     used     free   shared    buffers   cached
Mem:       7712     2742     4969        0        164     1838
-/+ buffers/cache:   740     6972 
Swap:         0        0        0

… and "df" is telling me that the filesystem is using 4.4GB. So, there should be enough remaining RAM to cache the entire filesystem.

Based upon that, I would think that the only I/O latency I experience should be limited to the time it would take to read the files into the cache the first time. After that, I wouldn't think it would need to fetch them again, and all writes could just be cached in the background, so I shouldn't have to wait for those at all. (Of course, the big wait would probably be when I wanted to shut down, and 4GB of dirty cache needed to be written to the flash).

But, that's not what I'm experiencing at all. Instead, everything is agonizingly slow. Switching the focus to other windows results in the all of the windows going "dim" for about 10 seconds (what does the window-manager need, for this task, from the disk, that it didn't surely write there? And if it wrote it earlier in the windowing session, why isn't in the cache?), and doing something like using apt-get to install something takes several minutes after the .deb file gets downloaded (how is that .deb not already in the cache? How are the new files from the package not being written to the cache? And it's not that the installer is having to go read /var/lib/dpkg/available or /var/lib/dpkg/status because this happens to each package even when I try to install a bunch at once).

I've tinkered with /proc/sys/vm/vfs_cache_pressure, /proc/sys/vm/swappiness, /proc/sys/vm/dirty_writeback_centisecs, /proc/sys/vm/dirty_ratio... all without any significant effect.

I'm clearly unclear on how Linux disk caching works. So, fill me in. Can the kernel be configured to (provided it's got enough RAM for it):

Read files from disk once and then keep them cached, and
Cache all writes and just get them written to the disk during idles (and if that's 20 minutes from now, fine)?

And, if it can do this, what am I missing?

Best Answer

USB thumbdrive raw performance

For starters I'd take a look at the USB thumb drive just to make sure that it's operating at an acceptable level. You can use the following 2 commands to measure it's performance:

$ hdparm -t /dev/sdb

$ dd count=100 bs=1M if=/dev/zero of=/media/disk/test oflag=sync

Substitute in the appropriate HDD info for your given setup.

File system

Also take a look at this article on increasing performance of USB thumbdrives from the Ubuntu help site.

References

Potential Method #1 - F_DROP_CACHES

I found a method from 2012 that discusses a proposed patch to the Linux kernel in this mail thread titled: Re: [RFC Patch] fs: implement per-file drop caches.

excerpt

Cong> This is a draft patch of implementing per-file drop caches.

Interesting. So can I do this from outside a process? I'm a SysAdmin, so my POV is from noticing, finding and fixing performance problems when the system is under pressure.
Cong> It introduces a new fcntl command  F_DROP_CACHES to drop  
Cong> file caches of a specific file. The reason is that currently  
Cong> we only have a system-wide drop caches interface, it could  
Cong> cause system-wide performance down if we drop all page caches  
Cong> when we actually want to drop the caches of some huge file.
How can I tell how much cache is used by a file? And what is the performance impact of this when run on a busy system? And what does this patch buy us since I figure the VM should already be dropping caches once the system comes under mem pressure...

Cong> Below is small test case for this patch:

The thread includes both a testcase and the actual patch to several files within the Linux kernel which adds an additional function to fs/drop_caches.c called drop_pagecache_file(struct file *filp). This function is then accessible through the frontend tool, fnctl.c via the command F_DROP_CACHES. This case calls this function:

file_drop_caches(filp, arg);

Which handles the dropping of all the caches associated with the given file. From the file include/linux/mm.h:

void file_drop_caches(struct file *filp, unsigned long which);

So this can be used?

I found no evidence that this patch ever made its way into the main Linux kernel code repository, so this option would appear to be available, only if you're willing to recompile the Linux kernel yourself.

Potential Method #2 - Using dd

In that same thread, another user mentions a completely different methodology that makes use of dd.

The following is excerpt from that email

This is useful functionality. Though isn't it already provided with POSIX_FADV_DONTNEED? This functionality was added to GNU dd (8.11) a year ago.
Here are the examples from that patch:
Advise to drop cache for whole file
 $ dd if=ifile iflag=nocache count=0
Ensure drop cache for the whole file
 $ dd of=ofile oflag=nocache conv=notrunc,fdatasync count=0
Drop cache for part of file
 $ dd if=ifile iflag=nocache skip=10 count=10 of=/dev/null
Stream data using just the read-ahead cache
 $ dd if=ifile of=ofile iflag=nocache oflag=nocache

Testing it out

I wasn't 100% positive how to test this out but I came up with the following approach.

make a 100MB file

$ dd if=/dev/urandom of=sample.txt bs=100M count=1

trace file accesses using fatrace
```
$ sudo fatrace | grep sample.txt
```
run top so we can monitor memory usage, note amount free.
```
$ top
```
open file, note amount of free memory now. Note the fatrace of the file sample.txt.
```
$ cat sample.txt > /dev/null
```

drop the file from memory, note amount of free memory now. Note the output of fatrace.

$ sudo dd of=/home/saml/tst/162600/sample.txt \
    oflag=nocache conv=notrunc,fdatasync count=0

Example

In terminal #1:

$ dd if=/dev/urandom of=sample.txt bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB) copied, 7.37996 s, 14.2 MB/s

$ ls -l sample.txt 
-rw-rw-r--. 1 saml saml 104857600 Oct 17 22:54 sample.txt

In terminal #2:

$ top
...
KiB Mem:   7968336 total,  6900956 used,  1067380 free,   267080 buffers
...

In terminal #3:

$ sudo fatrace | grep sample.txt

Now open the file, sample.txt, and note the amount of RAM. In terminal #1.

$ cat sample.txt > /dev/null

In terminal #2:

KiB Mem:   7968336 total,  7011896 used,   956440 free,   267336 buffers

Notice the output of fatrace in terminal #3:

cat(25940): R /home/saml/tst/162600/sample.txt
cat(25940): R /home/saml/tst/162600/sample.txt
cat(25940): RC /home/saml/tst/162600/sample.txt

Now remove the file from RAM, in terminal #4:

$ sudo dd of=/home/saml/tst/162600/sample.txt \
    oflag=nocache conv=notrunc,fdatasync count=0

Note the output of fatrace in terminal #2:

dd(26229): O /home/saml/tst/162600/sample.txt
dd(26229): CW /home/saml/tst/162600/sample.txt

Note the RAM in terminal #3:

KiB Mem:   7968336 total,  6908364 used,  1059972 free,   267364 buffers

So it would seem that all of the that was consumed by the file in RAM is freed.

Potential Method #3 - python-fadvise

Thanks to a comment by @frostchutz, there's another tool, a Python script, named [pyadvise][4] which provides a much simpler interface than the above dd methods. This script makes use of the same posix_fadvise(2) interface.

Example

$ sudo pyadvise --help
Usage: 
    pyadvise [options] [FILE]..

Options:
  -h, --help        show this help message and exit
  -w, --willneed    The specified files will be accessed in the near future
  -s, --sequential  The application expects to access the specified files
                    sequentially (with lower offsets read before higher ones)
  -d, --dontneed    The specified files will not be accessed in the near
                    future
  -r, --random      The specified files will be accessed in random order
  -o, --noreuse     The specified files will be accessed only once. Under
                    Linux, this operation is a no-op; see contrib/copyfileobj-
                    fadvise.py in the python-fadvise source tree for an
                    example on how to achieve approximately the same effect
  -n, --normal      Indicates that the application has no advice to give about
                    its access pattern for the specified files. If no advice
                    is given for an open file, this is the default assumption
  -v, --verbose     Explain what is being done

And if we repeat the above test and use pyadvise in place of dd:

$ pyadvise -d /home/saml/tst/162600/sample.txt

I noticed an identical drop in the RAM being consumed as before when I used dd.

Best Answer

USB thumbdrive raw performance

File system

References

Related Solutions

Linux Memory – Understanding Buffers Column in free Command Output

Linux – Drop a specific file from the Linux filesystem cache

Potential Method #1 - F_DROP_CACHES

Potential Method #2 - Using dd

Example

Potential Method #3 - python-fadvise

Related Question