Command-Line Terminal – Protocols and Standards Used

command linestandardterminal

I was wondering how the "GUI" of a command line application is communicated over a network. Most of the time, it's quite simple (plain text / input) but sometimes it's more complex (aptitude).

Is it defined by some sort of standard so that anyone can write their own terminal and that all terminal implementations behave in the same way (colors, positioning, etc.)?

Best Answer

Console programs typically use curses or one of its successors¹ to build the sorts of text user interfaces you're talking about.

These libraries use one of two databases, called termcap and terminfo.² These databases contain maps that tell the library what codes to send to get desired actions with a large number of diverse terminal types. The vast majority of the terminal types you'll find defined in these databases didn't survive the days of real terminals, and so are now only of historical interest.

ANSI Terminals

Modern Unix terminal emulators³ use the ANSI X3.64 protocol or one of its later variants:

  • ANSI X3.64: A standard for controlling "glass terminals" — as opposed to teletypes — it is based on special sequences of characters which the remote terminal interprets. For instance, if the Unix box wants to tell an ANSI X3.64 compatible terminal to move its cursor to the upper left corner of the screen, it sends the characters ESC [ 1 ; 1 H. The first two characters tell the terminal to expect a control sequence, the 1s are the row and column, and H is the command meaning "move cursor".⁴

    Trivia: many PC BBSes used ANSI codes, too. (Still do, actually.)

  • DEC VT100: The first really popular ANSI-compatible glass terminal was Digital Equipment Corporation's VT100. By proving ANSI's de jure standard out in the marketplace, it established a de facto standard that is still important today.

    Sometimes you see this called the VT102 protocol, that being a later cost-reduced — and therefore more popular — version of the VT100 plus all available expansion options built-in.

    The DEC terminal protocols are a backwards-compatible series, extending from the first ANSI-compatible model introduced in 1978 (the VT100) up through the VT500 series models produced by Boundless Technologies after they bought the terminal business from DEC in 1995. (Boundless is now out of business, but their terminals still pop up on the used market from time to time.)

  • xterm: A kind of amalgam of ANSI and the VT-whatever standards. Whenever you're using a GUI terminal emulator like xterm or one of its derivatives, you're usually also using the xterm terminal protocol, typically the more modern xterm-color or xterm-color256 variants.

  • Linux: The Linux console also uses an extended variant of the ANSI terminal protocol, in the same spirit as the xterm protocols. Most of its extensions have to do with the differences between a PC and a glass terminal. For example, the IBM keyboard has some keys not on a DEC VT-whatever. (And vice versa.)

    Some Unix systems have their own console terminal protocol, too. There's the scoansi ANSI X3.64 variant for SCO Unixes, for example.

A typical terminal emulator program is something of a mongrel, and doesn't emulate any single terminal model exactly. It might support 96% of all DEC VT escape sequences up through the VT320, yet also support extensions like ANSI color (a VT525 feature) and an arbitrary number of rows and columns. The 4% of codes it doesn't understand may not be missed if your programs don't need those features, even though you've told curses (or whatever) that you want programs using it to use the VT320 protocol. Such a program might advertise itself as VT320 compatible, then, even though, strictly speaking, it is not.⁵

Non-ANSI Terminals

There are a few other notable standards that you still sometimes come across:

  • Wyse: One of the earliest independent producers of glass terminals, Wyse started making terminals in the early 1980s before workstation computing began displacing minicomputers. Although Wyse terminals were able to emulate the VT100 and other popular terminal protocols, they also had their own native codes.

  • IBM 3270: Although this is not strictly a "Unix" terminal type, the need to connect Unix systems to IBM mainframes led to the creation of IBM 3270 series terminal emulator programs, which you can still find in use. Emulators for the later IBM 5250 series terminals are also fairly common, most often used these days for connecting to AS/400 and System i minicomputers.

  • Tektronix 4014: Before PCs and workstations largely displaced glass terminals and thus made bitmapped graphics a standard feature, there were expensive graphics terminals which drew graphics on the screen in response to text commands similar to the escape sequences described above. Probably the most popular of these was the Tektronix 4010 series.

    They were quite fun to use. You could write a program that drew a graphic, but then instead of simply running it to draw on your local terminal, you could redirect its output to a file:

    $ ./my4014program > my-neat-graphic

    You could then send that file to someone else, and they could cat it on their Tek terminal to see the graphic without having your program. Part of the charm was how slow these terminals were in drawing, so you could watch the graphic build up over several seconds.

Working With Unix Terminal Emulation Today

You can find out which terminal standard you're asking libraries like curses to use by looking at the TERM environment variable:

$ echo $TERM
xterm-color

When you ssh to another system, the TERM variable is carried along so the remote Unix box knows how to communicate with your local terminal.

Because so many of these protocols are ANSI X3.64 variants, and because the ubiquitous ASCII and UTF-8 character coding standards take care of so much else, an incorrect TERM variable isn't typically catastrophic. Things that tend to break are extended keys like Home and Page Up, Alt-whatever key combinations, and typographical display features like color, boldface, etc.

Footnotes:

  1. Most commonly, ncurses.

    There are also outright competitors to the curses API, such as S-Lang.

  2. AT&T promulgated terminfo as a replacement for BSD's termcap database, and it was largely successful in replacing it, but there are still programs out there that still use the old termcap database. It is one of the many BSD vs. AT&T differences you can still find on modern systems.

    My macOS box doesn't have /etc/termcap, but it does have /usr/share/terminfo, whereas a standard installation of FreeBSD is the opposite way around, even though these two OSes are often quite similar at the command line level.

  3. minicom, xterm, mintty, GNOME Terminal, Terminal.app, etc.

  4. Properly-written Unix programs don't emit these escape sequences directly. Instead, they use one of the libraries mentioned above, telling it to "move the cursor to position (1,1)" or whatever, and the library emits the necessary terminal control codes based on your TERM environment variable setting. This allows the program to work properly no matter what terminal type you run it on.

  5. Old text terminals had a lot of strange features that didn't get a lot of use by programs, so many popular terminal emulator programs simply don't implement these features. Common omissions are support for sixel graphics and double-width/double-height text modes.

    The maintainer of xterm wrote a program called vttest for testing VT terminal emulators such as xterm. You can run it against other terminal emulators to find out which features they do not support.

Related Question