Filesystems – Difference Between Distributed and Cluster File Systems

I'm using AFS, NFSv3, NFSv4, and CIFS currently. CIFS is primarily for supporting Windows clients and I find it less suitable for UNIX/Linux clients since it requires a separate mount and connection for each user accessing the share. Users can share the same mount point, but they will be seen as the same user on the server-side of the connection.

NFSv3 is primarily used by directories being exported to other UNIX/Linux servers since it's stable and simple to deal with. With both AFS and NFSv4 I am using Kerberos. Using NFSv4 on Ubuntu 8.04 and older I found it a bit unstable, but it has steadily improved and I have no stability issues with 10.04+. It does appear to be a performance bottleneck to use sec=krb5p so I tend to use sec=krb5i or sec=krb5.

One issue I have is how Kerberos tickets are handled with Linux's NFSv4 layer. A daemon periodically scans /tmp for files beginning with krb5cc_ and matches the ticket up with the file owner. If a user has more than one ticket they own under /tmp, it will use whichever ticket file is found first when scanning. I've accidentally changed my identity when temporarily acquiring a ticket for other purposes. AFS stores tickets in Kernel-space and are associated with a login session normally. I can login twice as the same Linux user, but still use different AFS credentials on each login without interference. I also have to explicitly load credentials into the kernel which normally happens automatically during login. I can safely switch tickets in userspace without interfering with file permissions.

Overall, I like many of the ideas of AFS better than NFSv3/4, but it does have quite a bit smaller of a community developing it compared to NFS and CIFS. It's also properly known as OpenAFS, AFS is the name of IBM's closed-source offering. A big difference between AFS and NFS is that AFS is more consistent in it's network protocol and support. AFS does provide locking in-band instead of using a side-band protocol like NFSv3. It also offers a more sophisticated ACL system in-between POSIX ACLs and NFSv4/NTFS/CIFS ACLs. This, unlike the POSIX ACL addition to NFSv3, is a standard part of it's protocol and both Windows and UNIX/Linux clients can access and modify them. It also doesn't suffer from the 16 group limit that many NFSv3 servers have. This makes AFS appear more consistent in my mind across Windows and UNIX systems. Also, since AFS is only accessible via it's network protocol, there aren't issues where the actual underlying filesystem behaves slightly differently from the exported view of it. For example, in Linux, a file may have MAC or SELinux labels controlling access or other extended attributes that aren't visible over NFS. AFS, on the other hand just doesn't have extended attributes.

I think you're confused, possibly because you've read several documents that use different terminology. Terms like “block size” and “cluster size” don't have a universal meaning, even within the context of filesystem literature.

Filesystems

For ext2 or ext3, the situation is relatively simple: each file occupies a certain number of blocks. All blocks on a given filesystem have the same size, usually one of 1024, 2048 or 4096 bytes. A file¹ whose size is between N blocks plus one byte and N+1 blocks occupies N+1 blocks. That block size is what you specify with mke2fs -b. There is no separate notion of clusters.

The FAT filesystem used in particular by MS-DOS and early versions of Windows has a similarly simple space allocation. What ext2 calls blocks, FAT calls clusters; the concept is the same.

Some filesystems have a more sophisticated allocation scheme: they have fixed-size blocks, but can use the same block to store the last few bytes of more than one file. This is known as block suballocation; Reiserfs and Btrfs do it, but not ext3 or even ext4.

Utilities

Unix utilities often use the word “block” to mean an arbitrarily-sized unit, typically 512 bytes or 1kB. This usage is unrelated to any particular filesystem or disk hardware. Historically, the 512B block did come about because disks and filesystems at the time often operated in 512B chunks, but the modern usage is just arbitrary. Traditional unix utilities and interfaces still use 512B blocks sometimes, though 1kB blocks are now often preferred. You need to check the documentation of each utility to know what size of block it's using (some have a switch, e.g. du -B or df -B on Linux).

In the GNU/Linux stat utility, the blocks figure is the number of 512B blocks used by the file. The IO Block figure is the preferred size for file input-output, which is in principle unrelated but usually an indication of the underlying filesystem's block size (or cluster size if that's what you want to call it). Here, you have a 13-byte file, which is occupying one block on the ext3 filesystem with a block size of 2048; therefore the file occupies 4 512-byte units (called “blocks” by stat).

Disks

Most disks present an interface that shows the disk as a bunch of sectors. The disk can only write or read a whole sector, not individual bits or bytes. Most hard disks have 512-byte sectors, though 4kB-sector disks started appearing a couple of years ago.

The disk sector size is not directly related to the filesystem block size, but having a block be a whole number of sectors is better for performance.

¹ _{Exception: sparse files save space .}