Ubuntu – Using mv & cp on a headless server (SSH timeout interferring?)

Usually this means you've forwarded the port to the wrong IP address on the LAN.

Your NAT router receives incoming traffic on port 57757, and sends it to a particular IP address and port on the LAN.

By default, Ubuntu doesn't filter incoming connection attempts with a firewall. So unless you've changed firewall settings in Ubuntu, an attempt to open a connection to any TCP port, 1 through 65535, will either:

• accept the connection, if the port is open
• reject the connection attempt, if the port is closed

If ports were filtered by a firewall, then you'd get what you're seeing--no response to a connection attempt. But:

• unless you've changed firewall settings (e.g., ufw, iptables), no ports are filtered, and
• in any case, you can connect to port 22 on the LAN, so it's open.

When you forward a port to a nonexistent machine with a NAT router, it sends incoming traffic on that port to the nonexistent machine, which is to say that it sends it into a black hole; it is effectively dropped.

That causes exactly the situation you've described. So you can most likely fix this by making sure the port is being forwarded to the correct IP address on the LAN.

If that turns out not to have been the problem...

...then you'll have to do some troubleshooting.

1. Is the port specified for the LAN side correct? That is, assuming you haven't changed the SSH server's configuration, is port 57757 on the WAN side set to forward to port 22 on the OpenSSH server? (You may want to double-check this.)

2. Maybe there's some problem with the specific port you've chosen (57757). Try a different one and see if that works better.

   (If it doesn't, and you proceed with these instructions, either change it back or replace "57757" below with the new number.)

3. Try restarting the OpenSSH server. That may help if there is a network problem. If that doesn't help, try restarting the router and cable/DSL/ISDN modem too.

   If for some reason you cannot restart all three devices, I recommend restarting whatever you can. If you can't restart the OpenSSH service, at least you can restart the service and (more likely to fix this) take the interface down and bring it up again.

   To restart the OpenSSH server:

   sudo restart ssh
   

   To bring the network interface down, first figure out what interface it's on:

   ifconfig
   

   Typically, for a machine with a single Ethernet card and/or a single wireless card, the Ethernet is eth0 and the wireless is wlan0.

   If the wired Ethernet connection is what you want to take down and restart, run:

   sudo ifdown eth0
   

   Then run:

   sudo ifup eth0
   

   Alternatively, you can run:

   sudo ifconfig eth0 down
   

   Followed by:

   sudo ifconfig eth0 up
   

   If the machine is using NetworkManager to manage the interface on which the OpenSSH server is running, I still recommend trying the above ways, but you can also try disconnecting and reconnecting in NetworkManager.

   For an Ethernet connection, also try unplugging the cable and plugging it back in. For a wireless connection, try turning it off with the hardware switch (if there is one), and back on again.

   Something strange is going on here and none of this takes very long--it's worthwhile to be thorough before undertaking more painstaking troubleshooting steps.

4. How are you attempting to access it from the WAN side? If you're using a machine on the LAN to do this (just connecting from the LAN to your router's WAN IP), this is only supported for some routers. After all, a router's job is to route traffic between the WAN and LAN sides, not to route traffic from one side to itself. Support for connecting to forwarded ports on the WAN IP from inside the LAN is actually the exception rather than the rule, though many home/office routers do have this feature.

   So if you're not testing the port forward from a host on the WAN side, you should do so. Your options for this are:

   • Connect yourself from the WAN side. This works if you have access to a machine there, for example, SSH access to a remote machine at school, work, a friend's house, or similar.

   • Connect the test machine between the router and whatever provides its Internet connection. If you have a cable/DSL/ISDN modem with an Ethernet port and your router is plugged into that, you can connect a switch to the modem and connect the router to the switch. Connect a computer to the switch. First see if that machine just gets Internet access--these days, many ISP's provide two or more separate IP addresses. If it doesn't, go to your router's setup page and check its WAN IP and WAN subnet mask, then statically assign an IP address to the switch-connected machine that is within the same subnet.

     This method has some disadvantages. It's a pain! Also, it's theoretically possible for an ISP to configure their network incorrectly so that the test machine connected to the switch could access the Internet. (Unless it's your ISP's intention to let you connect with more than one WAN IP and you happen to have picked a WAN IP for the test machine that your ISP has assigned to you, traffic between it and a true WAN host should be blocked/dropped by the ISP. But some ISP's have weird practices, so who knows?) If this happens, it won't likely cause anyone serious problems (and even if it did, you only have it connected for up to a few minutes). However, it could potentially be seen as an attempt to obtain additional access beyond the bounds of your subscription, and--more importantly--if another user has that same IP, it could intefere with their connection. Therefore, if you want to try this method, don't try to access the Internet from the test machine, stop immediately if you find the test machine can access the Internet, and don't attempt this at all if it's prohibited or advised against by your ISP. (And don't use this if the WAN side of your router is an office LAN, without consulting your network administrator first. That's not an ISP and there's no assumption that resources are provisioned to prevented undesired access.)

     There is a variation on this technique that is sometimes more appropriate. Your router probably gets its connection information--IP address, subnet mask, the IP address of the gateway (router) on the WAN that it uses when it doesn't know where to send something, and information about DNS servers--from your ISP, via DHCP, through the cable/DSL/ISDN modem. This is why you have to have the router plugged into the modem to give it the necessary configuration to make the results of WAN-side testing meaningful. But the router will typically remember this information, so long as it is actually connected to a network on the WAN side. So you can connect the router, modem, and test machine, but then, quickly, and before doing anything with the test machine besides making sure the switch sees it as connected, disconnect the modem.

   • Use an free service on the Internet to test your ports. Since inserting a test machine between your router's WAN interface and the Internet (above) is highly involved--and since it will show a port as accessible even if it's inaccessible due to being blocked by your ISP (which is also true of connecting to the router's WAN IP from the LAN side)--it's usually better to use a web-based port scanning service.

     There are many port scan services. (Some sport the phrase "check your firewall" with the idea that most people are trying to block rather than facilitate access.) This is one. If you choose to use that one, click Proceed, type 57757 into the text box, and click Use Specified Custom Port Probe. For the purposes of getting a server to run, you want it to be "open." "Closed" means the port is accessible but the server is not running (and thus the connection attempt was rejected). "Stealth" means the port was inaccessible--it's as if no machine was located there (or as if the port were forwarded where there's no machine).

5. OK, so you've determined it really isn't accessible from the Internet. Potentially you can scan it (ideally from the WAN side) to get details, though often this won't give helpful information.

   If you want to do this, then on the WAN side, you can run:

   sudo nmap -sS -sV -p57757 -vv WAN-IP

   If the port is shown as filtered, that confirms that packets sent there are probably going nowhere (or are blocked/dropped along the way).

6. It's worth checking to see if the problem is arising from the port exposed to the WAN being different from the port the server is actually listening on. Forwarding port 55757 on the WAN to port 22 on the LAN machine should make things work out fine, but perhaps somewhere (the server, the client) something is assuming the port number is the same from the server and client's perspective.

   Presumably you cannot forward port 22 through the router. Perhaps your ISP blocks that port. But if you can do that, do that!

   Otherwise, you can make the OpenSSH server actually listen on port 57757.

   To do this, back up the server configuration file:

   cd /etc/ssh
   sudo cp sshd_config sshd_config.old
   

   Then edit it:

   gksu gedit sshd_config
   

   Or use a console text editor if the machine doesn't have a GUI:

   sudo nano -w sshd_config
   

   Near the top of the file, this block of text appears:

   # What ports, IPs and protocols we listen for
   Port 22
   # Use these options to restrict which interfaces/protocols sshd will bind to
   #ListenAddress ::
   #ListenAddress 0.0.0.0
   Protocol 2
   # HostKeys for protocol version 2
   HostKey /etc/ssh/ssh_host_rsa_key
   HostKey /etc/ssh/ssh_host_dsa_key
   HostKey /etc/ssh/ssh_host_ecdsa_key
   #Privilege Separation is turned on for security
   UsePrivilegeSeparation yes
   

   Just change the Port 22 line at the top, to say Port 57757 instead.

   • You could add a port rather than changing it. I recommend testing with the simplest effective configuration, though.

     See man sshd_config for more details about configuring the OpenSSH server.

   Save the file, quit the text editor, and restart the SSH server with:

   sudo restart ssh
   

   Now change the port forward on the router so port 57757 forwards to port 57757 (not 22) on the OpenSSH server, and see if it's accessible from the Internet.

7. If it's still not working, see if maybe Ubuntu's firewall actually is blocking traffic originating from outside the LAN.

   (This is unlikely if you didn't configure it this way yourself, but if all your settings are right and none of the above steps revealed anything about the problem, it's worth checking.)

   Run:

   sudo iptables -L
   

   By default, in Ubuntu, the output looks like:

   Chain INPUT (policy ACCEPT)
   target     prot opt source               destination         
   
   Chain FORWARD (policy ACCEPT)
   target     prot opt source               destination         
   
   Chain OUTPUT (policy ACCEPT)
   target     prot opt source               destination
   

   This is a simple permissive policy, essentially equivalent to not running a firewall. (Indeed, if the netfilter firewall module weren't compiled into your kernel, your system would behave the same way as with the above settings, although the iptables command, which queries netfilter's settings, wouldn't work of course.)

   If your configuration doesn't look like that, read man iptables to figure out what they're doing, and/or edit your question (or, if you're a different person with a similar problem reading this, post a new question) to include them. Please note that, potentially, your iptables rules could disclose sensitive information about your configuration. Practically speaking, this is usually not the case--with the possible exception of rules about specific hosts that are blocked, or if your configuration is very bad/insecure--typically the usefulness of this information to an attacker, especially for a machine on a home/office LAN behind a NAT router, is minimal.