Linux – FILE size limitation according to Robert Love’s textbook

fileslimitlinuxsystem-programming

From Robert Love's Linux System Programming (2007, O'Reilly), this is what is given in the first paragraph (Chapter 1, Page 10):

The file position’s maximum value is bounded only by the size of the C type used to store it, which is 64-bits in contemporary Linux.

But in the next paragraph he says:

A file may be empty (have a length of zero), and thus contain no valid bytes. The maximum file length, as with the maximum file position, is bounded only by limits on the sizes of the C types that the Linux kernel uses to manage files.

I know this might be very, very basic, but is he saying that the file size is limited by the FILE data type or the int data type?

Best Answer

He's saying it's bound by a 64-bit type, which has a maximum value of (2 ^ 64) - 1 unsigned, or (2 ^ 63) - 1 signed (1 bit holds the sign, +/-).

The type is not FILE; it's what the implementation uses to track the offset into the file, namely off_t, which is a typedef for a signed 64-bit type.¹ (2 ^ 63) - 1 = 9223372036854775807. If a terabyte is 1000 ^ 4 bytes, that's ~9.2 million TB. Presumably the reason a signed type is used is so that it can hold a value of -1 (for errors, etc), or a relative offset.

Functions like fseek() and ftell() use a signed long, which on 64-bit GNU systems is also 64-bits.

^{1. See types.h and typesizes.h in /usr/include/bits.}

Related Solutions

POSIX Compliance – Is Linux ARG_MAX Different from Other System Variables?

There is no standard way to retrieve the list of configuration variables that are supported on a system. If you program for a given POSIX version, the list in that version of the POSIX specification is your reference list. On Linux, getconf -a lists all available variable.

fpathconf isn't specific to PATH. It's about variables that are related to files, which are the ones that may vary from file to file.

Regarding ARG_MAX on Linux, the rationale for depending on the stack size is that the arguments end up on the stack, so there had better be enough room for them plus everything else that must fit. Most other implementations (including older versions of Linux) have a fixed size.

Most limits go together with resource availability, with different resources depending on the limit. For example, a process may be unable to open a file even if it has fewer than OPEN_MAX files open, if the system is out of memory that can be used for the file-related data.

Linux is POSIX-compliant on this point by default, so I don't know where you're getting at.

If you use ulimit -s to restrict the stack size to less than ARG_MAX, you're making the system no longer compliant. A POSIX system can typically be made non-compliant in any number of ways, including PATH=/nowhere (making all standard utilities unavailable) or rm -rf /.

The value of ARG_MAX in limits.h provides a minimum that applications can rely on. A POSIX-compliant system is allowed to let execve succeed even if the arguments exceed that size. The guarantee related to ARG_MAX is that if the arguments fit in that size then execve will not fail due E2BIG.

File Size Limit

File size is limited by filesystem type not by OS. Typically, OS supports several filesystems, so there is no such thing like "OS file size limit". There are limits for well-known filesystems:

FAT32 - 4Gib  
NTFS - 16Eib  
ext2/3 - 16Gib - 2Tib (depends from block size)  
ext4 - 16Gib - 16Tib  
XFS - 9Eib  
ZFS - 16Eib

Best Answer

Related Solutions

POSIX Compliance – Is Linux ARG_MAX Different from Other System Variables?

File Size Limit

Related Question