Overwrite multiple lines of stdout at the command-line without losing terminal scrollback

Some terminals, such as xterm, support what is known as an “alternate screen”: there are separate screens for full-screen programs and for scrolling programs. In xterm, you can switch between the two screens with the “show alternate screen” command at the bottom of the Ctrl+mouse 2 menu.

This behavior is disabled by default in screen but can be enabled with the altscreen option: add altscreen on to your ~/.screenrc.

This is a bit of a complicated question. I'll try to answer your questions in turn, but first a general description:

The scrollback buffer is implemented by your terminal emulator (xterm, Konsole, GNOME Terminal). It contains all the text that has been displayed on the screen, including both standard output and standard error from every program you run in the terminal. It's entirely terminal functionality to let you look at past output that may have scrolled past you or to check on what something said earlier.

You can think of the scrollback buffer as a long page of logged output and your terminal window as a window looking at just part of it at any one time. If you haven't scrolled up any, what you're looking at is the tail end of the buffer. Usually there will be a limit configured in the terminal of how many lines it keeps track of before it starts to forget.

Suppose that limit is 1000 lines. For the first thousand lines of output in your session you just append to the buffer, and you can scroll up right to the start of your session. As soon as get the 1001st line of output the first line in the buffer is erased, and the furthest back you can scroll will be the second line of your session. The buffer will always contain the most recent thousand lines of output that were displayed on your screen, and you can scroll up to look at earlier output at any time.

• Does it mean "function" as in "subroutine" or as in "feature"?

  This is "function" as in "feature". The terminal emulator has a functionality that records what's on the screen and lets you scroll up and down in it. The consoles on some systems also support limited scrollback.

  It gets a bit more complicated once you throw screen into the mix. At that point, screen is emulating the scrollback buffer itself - that's why you can copy and paste from it within the program, rather than only with (say) X selection.

• Is there is a Unix standard or API for this scrollback buffer?

  The short answer is no, it's just provided by your terminal. The longer answer we'll get to at the bottom.

• In a "stack" of programs, such as vim launched in screen launched in bash launched in ssh launched in a terminal emulator, which of these programs are controlling the scrollback buffer?

  In the case of vim and bash, they aren't controlling it at all (caveat, again, below). Your terminal provides the scrollback buffer for all the programs inside it, starting from your shell. screen, as mentioned above, is simulating scrollback itself.

• I also used screen to dump the scrollback to a file. This file had a lot of white space at the top, and it seems the "view" my terminal emulator shows me is simply the bottom few lines of the of the buffer.

  This is screen's internal buffer. What's on your screen at the time will generally be what's at the very bottom of the buffer.

• Is this why a program like vim can "clear" my entire terminal window, because it gets temporary access to the parent shell's scrollback buffer?

  Here is one part of where it gets much more complex. Virtually all X-based terminal emulators are simulating a VT100, and one thing they do there is support an "Alternate Screen Buffer". Unlike the ordinary buffer that's used for most terminal interaction with sequential output, the alternate screen buffer is just the exact size of your terminal. There's no scrolling up or down in it because it's no bigger than what's displayed.

  The idea there is to allow a full-screen application to do what it needs to do without being interfered with by anything you already had on the screen, and then to let you go back to exactly the display you had before. That's why when you enter vim it fills the whole screen, but when you leave it then the terminal output you had beforehand — all your past prompts and command output — comes back again. vim switches into the alternate screen buffer when it starts and back to the normal buffer when it exits.

  This alternate buffer is one of the caveats I mentioned above. Sometimes, the program really does have the ability to tell the terminal what to do with the buffer.

  screen is another program that does this, which is why your terminal's scrolling functionality generally doesn't work while you're in a screen session — screen emulates the scrollback buffer itself, so you have to use its internal functionality to get to old output.

• Or does vim use its own scrollback buffer that is somehow overlaid on top of the parent scrollback buffer?

  I've mostly answered this in the previous question, but the short answer to this particular question is that vim does get its own temporary buffer, with no scrollback, from the terminal, and then performs all its own scrolling of your documents internally.

All those exceptions I mentioned:

It gets slightly more complicated again. I said that applications don't have any control over the scrollback and that it's provided entirely by the terminal. In some cases, with some terminals, there is limited interaction. The program prints out certain escape sequences — if you've ever used terminal colouring manually in the past you'll have seen what they look like — and the terminal can interpret those and change its behaviour, or even send back information to the program. Which escape sequences are available are described in the termcap (terminal capability) database.

Some terminals do support limited querying and manipulation of the scrollback buffer. Many xterm derivatives have escape sequences that direct the terminal to scroll its view. Many terminals also support specifying a particular area of the screen to scroll, leaving all the rest intact. That tends to break the scrollback buffer.

Almost all terminals support sequences to move the cursor around the screen, which is how the ncurses library is able to update all different parts of the display. You can look at the VT100 sequences supported by xterm. The way that these interact with the scrollback buffer can be a bit odd at times, particularly in the case of something that implements its own scrolling behaviour, like the less command. You can end up with duplicated or missing lines in your scrollback because less redrew text over the top in a way your terminal didn't expect. Other programs sometimes end up filling your buffer with multiple copies of their entire display.