POSIX Text Files – Conditions for a File to be a Text File as Defined by POSIX

filesposixtext;

POSIX defines a text file as:

A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character. Although POSIX.1-2017 does not distinguish between text files and binary files (see the ISO C standard), many utilities only produce predictable or meaningful output when operating on text files. The standard utilities that have such restrictions always specify "text files" in their STDIN or INPUT FILES sections.

Source: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_403

However, there are several things I find unclear:

Must a text file be a regular file? In the above excerpt it does not explicitly say the file must be a regular file
Can a file be considered a text file if contains one character and one character only (i.e., a single character that isn't terminated with a newline)? I know this question may sound nitpicky, but they use the word "characters" instead of "one or more characters". Others may disagree, but if they mean "one or more characters" I think they should explicitly say it
In the above excerpt, it makes reference to "lines". I found four definitions with line in their name: "Empty Line", "Display Line", "Incomplete Line" and "Line". Am I supposed to infer that they mean "Line" because of their omission of "Empty", "Display" and "Incomplete"- or are all four of these definitions inclusive as being considered a line in the excerpt above?

All questions that come after this block of text depend on inferring that "characters" means "one or more characters":

Can I safely infer that if a file is empty, it is not a text file because it does not contain one or more characters?

All questions that come after this block of text depend on inferring that in the above excerpt, a line is defined as a "Line", and that the other three definitions containing "Line" in their name should be excluded:

Does the "zero" in "zero or more lines" mean that a file can still be considered a text file if it contains one or more characters that are not terminated with newline?
Does "zero or more lines" mean that once a single "Line" (0 or more characters plus a terminating newline) comes into play, that it becomes illegal for the last line to be an "Incomplete Line" (one or more non-newline characters at the end of a file)?
Does "none [no line] can exceed {LINE_MAX} bytes in length, including the newline character" mean that there a limitation to the number of characters allowed in any given "Line" in a text file (as an aside, the value of LINE_MAX on Ubuntu 18.04 and FreeBSD 11.1 is "2048")?

Best Answer

Must a text file be a regular file? In the above excerpt it does not explicitly say the file must be a regular file

No; the excerpt even specifically notes standard input as a potential text file. Other standard utilities, such as make, specifically use the character special file /dev/null as a text file.
Can a file be considered a text file if contains one character and one character only (i.e., a single character that isn't terminated with a newline)?

That character must be a <newline>, or this isn't a line, and so the file it's in isn't a text file. A file containing exactly byte 0A is a single-line text file. An empty line is a valid line.
In the above excerpt, it makes reference to "lines". I found four definitions with line in their name: "Empty Line", "Display Line", "Incomplete Line" and "Line". Am I supposed to infer that they mean "Line" because of their omission of "Empty", "Display" and "Incomplete"

It's not really an inference, it's just what it says. The word "line" has been given a contextually-appropriate definition and so that's what it's talking about.
Can I safely infer that if a file is empty, it is not a text file because it does not contain one or more characters?

An empty file consists of zero (or more) lines and is thus a text file.
Does the "zero" in "zero or more lines" mean that a file can still be considered a text file if it contains one or more characters that are not terminated with newline?

No, these characters are not organised into lines.
Does "zero or more lines" mean that once a single "Line" (0 or more characters plus a terminating newline) comes into play, that it becomes illegal for the last line to be an "Incomplete Line" (one or more non-newline characters at the end of a file)?

It's not illegal, it's just not a text file. A utility requiring a text file to be given to it may behave adversely if given that file instead.
Does "none [no line] can exceed {LINE_MAX} bytes in length, including the newline character" mean that there a limitation to the number of characters allowed in any given "Line" in a text file

Yes.

This definition is just trying to set some bounds on what a text-based utility (for example, grep) will definitely accept — nothing more. They are also free to accept things more liberally, and quite often they do in practice. They are permitted to use a fixed-size buffer to process a line, to assume a newline appears before it's full, and so on. You may be reading too much into things.

Examples

First we'll show what octal dump (od) shows when we tell it to dump the contents of file a.txt in a hexidecimal format (-x).

$ od -x a.txt 
0000000 3737 2036 6572 6f62 746f 370a 3737 0020
0000020 0000 0000 0000 0000 0000 0000 0000 0000
*
0000140 0000 0000 0000 0000 0000 0000 0000 6567
0000160 6e61 2079 652f 6374 662f 7473 6261 380a
0000200 3332 7220 6265 6f6f 2074 380a 3432 0020
0000220 0000 0000 0000 0000 0000 0000 0000 0000
*
0000320 7865 7469 000a
0000325

We can use hexdump to do something similar, showing the data in hexidecimal format, however it will also show the value as an ASCII character if possible.

hexdump

$ hexdump -C a.txt 
00000000  37 37 36 20 72 65 62 6f  6f 74 0a 37 37 37 20 00  |776 reboot.777 .|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 67 65  |..............ge|
00000070  61 6e 79 20 2f 65 74 63  2f 66 73 74 61 62 0a 38  |any /etc/fstab.8|
00000080  32 33 20 72 65 62 6f 6f  74 20 0a 38 32 34 20 00  |23 reboot .824 .|
00000090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000000d0  65 78 69 74 0a                                    |exit.|
000000d5

Looking at the above output you'll notice several sequences of 00 00 00. These are those ^@ characters you were asking about originally.

Incidentally the character 00 is the null character.

Vi – Does Vi Silently Add a Newline (LF) at the End of File?

This is the expected vi behavior.

Your file has an incomplete last line so strictly speaking (i.e. according to the POSIX standard), it is not a text file but a binary file.

vi which is a text file editor, not a binary one, gracefully fixes it when you save it.

This allows other text file tools like wc, sed and the likes to provide the expected output. Note that vi isn't silent about the issue:


$ printf "one\ntwo" >file     # Create a unterminated file
$ cat file                    # Note the missing newline before the prompt
one
two$ wc -l file               # wc ignores the incomplete last line
       1 file
$ sed '' file > file1
$ cat file1                   # so does a legacy sed
one
$ PATH=$(getconf PATH) sed  '' file
one                           # while a POSIX conformant sed warns you:
sed: Missing newline at end of file file.
two
$ vi file
one
two
~
~
~                             # vi tells you too about the issue
"file" [Incomplete last line] 2 lines, 7 characters

:w

"file" 2 lines, 8 characters  # and tells it writes two lines
                              # You'll even notice it writes one more
                              # character if you are a very shrewd observer :-)
:q
$ cat file                    # the file is now valid text
one
two
$ wc -l file                  # wc reports the expected number of lines
       2 file
$ sed '' file > file1         # sed works as expected
$ cat file1
one
two

Note, to get some clues about what vi version you are running, you can use the :ve command. It shows here I'm using a legacy SVR4 one here, definitely not vim:

:ve
Version SVR4.0, Solaris 2.5.0

Apparently, yours is stating:

:ve
Version 3.10

That likely means AIX vi is based on SVR3 source code.

In any case, this behavior, and the [Incomplete last line] warning message have been in the legacy Bill Joy's vi source code since at least 1979 and AFAIK, retained in all branches created from System V source code releases, from which proprietary Unix like AIX were built.

Chronologically speaking, this behavior is then not a consequence of POSIX conformance but more a consequence of Bill Joy's original decision to be helpful with users editing bogus text files, and then, a decade later, the POSIX committee decision to keep this tolerance.

If you use ed instead of vi, you'll notice that the former is more verbose about the issue, at least if your ed is from SVR3 or newer source branch:

$ ed file
'\n' appended
8
q

Note also that an empty file is a valid text file that happens to contain zero lines. As there is then no unterminated line to fix, vi doesn't append a newline when saving the file.

Best Answer

Related Solutions

What does ^@^@^@ mean in a text file

Examples

Vi – Does Vi Silently Add a Newline (LF) at the End of File?

Related Question