SSH Output – Why It Isn’t Line Buffered

That's an optimisation done by screen.

When you type echo<Cr> in screen. Because of the local echo and the icrnl and onlcr settings of the pseudo terminal device in the screen window, the \r\n sequence is sent to master side (to screen).

screen implements a terminal emulator where \r is meant to bring the cursor to the beginning of the line and \n to move the cursor down. To do that, where a terminal emulator like xterm would do X API calls to move the cursor to the beginning of the line, screen has to send escape codes to the host terminals it is attached to to tell it/them to move the cursor to the left hand side of the screen window.

In case you've split the window vertically, that means send cursor positioning escape sequences to wherever the left hand side of the screen windows is. If not or if on the left hand side of the host terminal, screen would just pass those \r and \n characters along so that the cursor be moved to the beginning of the line and one line down on the host terminal as well (since all terminals treat \r and \n the same in that instance).

Now, echo runs and outputs a \n character. Because of onlcr again in the screen window tty, screen receives \r\n again. \r tells it to move to the beginning of the line, but the cursor is already at the beginning of the line, so no need to do anything which is why the host terminal doesn't receive a second \r character. Then to move the cursor down because of the \n it receives, screen sends \n to the host terminal.

You can verify that by running in screen:

printf '\r\r\r\r'

You'll notice that screen only sends one \r character to its host terminal.

A Very Disappointing Self-Answer

Having set this problem aside for a day and come back to it, I was both relieved and perturbed (more perturbed than relieved) to find that everything was, mysteriously, working properly.

So, What Was the Issue?

No settings were changed or adjusted -- not on the router, not on the SSH server, and not on the SSH client's machine. It's fairly safe to say it was the router not handling the incoming traffic properly, in spite of proper settings. Given that dinky home router software isn't really designed to deal with port forwarding, it took the poor guy a while to implement the necessary changes.

But It's Been Like 6 Hours!!

Yeah dude, I know. I spent all day trying to figure out what was wrong -- and didn't ever find it because there wasn't anything wrong. Evidently, it can take 6 hours -- possibly more -- for the router settings to take effect.

So How Do I Know If This Is My Issue?

A nifty tool I came across during this escapade is tcpdump. This lean little guy sniffs traffic for you, offering valuable insight into what's actually going on. Plus, he's got some super filtering features that allow you to narrow down exactly what you want to look at/for. For example, the command:

tcpdump -i wlan1 port 22 -n -Q inout

Tells tcpdump to look for traffic via the wlan1 interface (-i = 'interface'), only through port 22, ignore DNS name resolution (-n = 'no name resolution'), and we want to see both incoming and outgoing traffic (-Q accepts in, out, or inout; inout is the default).

By running this command on your SSH server while attempting to connect via a remote machine, it quickly becomes clear where precisely the problem lies. There are, essentially, 3 possibilities:

1. If you're seeing incoming traffic from the remote machine, but no outgoing traffic from your local server, the problem lies with the server: there's probably a firewall rule that needs to be changed, etc.
2. If you're seeing both incoming and outgoing, but your remote machine isn't receiving the response, it's most likely the router: it's allowing the incoming traffic, but dropping your outgoing packets.
3. If there's no traffic at all, that's probably a router issue as well: the remote machine's SYN packets are being ignored and dropped by the router before they even reach your server.

And once you've discovered where the problem lies, a fix is (usually) trivial.