How to PXE boot over WAN

I currently have a Debian PC and a Raspbian Raspberry Pi on my home network behind my router. The router has a port open for me in which I can ssh into the rPi when I'm away, and then once I'm in the network, I can do whatever I want on the home network.

While on travel, my Debian PC's hard drive appears to have died; at the very least it is refusing to boot up.

I did not leave behind any boot CDs for anyone to insert as I have always booted the box over PXE, with my Tomato home router configured to point to a VM on my laptop to serve the BOOTP and TFTP images needed to boot any Linux installer, System Rescue CD, or Trinity Rescue Kit.

That laptop is now with me on travel.

Can I set up SSH port forwarding from my laptop when I connect to my home rPi and temporarily reconfigure the IP address to [I think] dhcp-boot such that my Debian PC can PXE boot using my remote laptop VM over the Internet?

UPDATE 3 Aug: Following Matt's suggestion below was a good start but still not complete. I'm still no further along than before.

The TFTP server listens on UDP port 69, but it appears using the -R option to ssh only forwards TCP.

This and this provided one way to proceed, setting up pipes around the tunnel to translate the UDP traffic to TCP before the tunnel and TCP back to UDP after the tunnel, but comments indicated this isn't the best way. Also, this did not work for me.

Using socat instead of the pipes seemed to produce less controversy, so I thought I would try that, but this didn't work either.

To capture what I've done so far, I tried to replicate the problem locally on my laptop as follows.

I already have my VM running the TFTP server on port 69 that can at the very least serve a pxelinux.0 file.

I created a new VM booting a System Rescue CD ISO image. When I booted this VM, I set it up as a DHCP server:

# vi /etc/dhcp/dhcpd.conf

In the file, I specified the following:

subnet 192.168.1.0 netmask 255.255.255.0 {
   range 192.168.1.251 192.168.1.253;
   option routers 192.168.1.1;
   filename "pxelinux.0";
   next-server 192.168.1.252;
}

Then, I restarted the DHCP server:

# /etc/init.d/dhcpd restart

Note: I now have two DHCP servers on this "foreign" network–the router that provided my laptop its IP address, and now this VM until it gets powered down. This introduces a race condition as my PXE-booting VM might receive a response from either DHCP server; I'm hoping my VM's response is the first to reach the PXE VM so that it can get the next-server directive above.

In practice with my limited exercises, it appears to not be an issue, but I don't want to leave my VM up for too long so I don't disturb anyone else over here.

As stated, with the DHCP server running, I booted my third diskless VM set up for PXE booting to exercise the PXE boot process.

The .252 host was my PXE server, so the PXE-boot VM was able to reach it, get the files needed via TFTP, and began booting as usual.

With the nominal case working, I decided to introduce the SSH tunnel into the mix as Matt suggested.

# vi /etc/dhcp/dhcpd.conf

I modified next-server to the IP address of the System Rescue CD VM itself:

subnet 192.168.1.0 netmask 255.255.255.0 {
   range 192.168.1.251 192.168.1.253;
   option routers 192.168.1.1;
   filename "pxelinux.0";
   next-server 192.168.1.13;
}

Then, I restarted the DHCP server:

# /etc/init.d/dhcpd restart

I also verified no TFTP server was running on this box:

# netstat -an | grep 69

Now my PXE-boot VM failed to boot. It received an IP address from the DHCP server but was not able to retrieve any TFTP boot image.

I brought over my PXE server's public key and placed it into the root authorized_keys file.

From the PXE server VM, I created my tunnel:

$ ssh -R 69:localhost:69 -R 2049:localhost:2049 root@192.168.1.13

Note I didn't need to forward port 2049 at this time, but I will need to do so later for NFS.

Same problem when booting the PXE VM: Couldn't get the image over TFTP.

The TFTP server was listening on UDP port 69:

udp        0      0 0.0.0.0:69              0.0.0.0:*

However, the forwarded port is listening on TCP:

tcp        0      0 127.0.0.1:69            0.0.0.0:*               LISTEN
tcp6       0      0 ::1:69                  :::*                    LISTEN

Tried the initial approach of using pipes. On the System Rescue CD VM:

# mkfifo /tmp/fifo
# nc -v -l -u -p 6963 < /tmp/fifo | nc -v -u 127.0.0.1 6964 > /tmp/fifo

Meanwhile, on the PXE server VM:

$ ssh -R 6964:localhost:6965 root@192.168.1.13
$ nc -v -l -p 6965 < /tmp/fifo | nc -v -u localhost 69 > /tmp/fifo

From the System Rescue CD VM, I tried to manually transfer the file:

# tftp localhost 6963 -c get pxelinux.0
connect to [127.0.0.1] from sysresccd.gentoo [127.0.0.1] 51072
too many output retries : Broken pipe

This still didn't work, so I tried again taking the tunnel out of the equation.

System Rescue CD VM:

# nc -v -l -u -p 6963 < /tmp/fifo | nc -v -u 127.0.0.1 6965 > /tmp/fifo

PXE server VM:

$ nc -v -l -p 6965 < /tmp/fifo | nc -v -u localhost 69 > /tmp/fifo

Transfer on System Rescue CD VM:

# tftp localhost 6963 -c get pxelinux.0

This still didn't work, so I cleaned up with rm /tmp/fifo on both boxes.

Next, I proceeded with the socat attempt, but that had similar results. I took the SSH tunnel out of the equation there as well, with same results.

This time, I couldn't use the System Rescue CD VM since it did not come with socat installed, so I used a different VM.

First, I wanted to verify the actual TFTP server was still available:

$ tftp 192.168.1.252
tftp> get pxelinux.0
tftp> quit

Sure enough, I got a nonzero-byte file downloaded.

This did not work when I used a box with nothing listening on the TFTP port:

$ tftp localhost
tftp> get pxelinux.0
Transfer timed out.

tftp> quit
boots$ rm pxelinux.0

Time to put socat into the mix.

On the PXE server:

$ ssh -p 22 -R 6969:localhost:6968 192.168.1.7
$ sudo socat UDP4-LISTEN:69,fork TCP4:localhost:6969

In another window on the PXE server:

$ socat -T10 TCP4-LISTEN:6968,fork UDP4:localhost:69

This still did not work:

$ tftp localhost
tftp> get pxelinux.0
Transfer timed out.

tftp> quit
boots$ rm pxelinux.0

Update 4 Aug: I think I found my problem but I'm not sure how to get past it.

I've made my setup even more basic. I eliminated the SSH tunnel altogether and stuck to straight UDP ports.

On my local TFTP client, simply this:

$ sudo socat -d -d -d -d udp4-recvfrom:69,reuseaddr,fork udp:192.168.1.252:69

Reminder that the .252 box is the TFTP server.

This still did not work:

$ tftp localhost
tftp> get pxelinux.0
Transfer timed out.

tftp> quit
boots$ rm pxelinux.0

So I ran Wireshark on both the .7 and the .252 boxes looking at UDP traffic and in particular TFTP traffic.

Here are my capture filters on the two Wireshark instances:

• .7: udp and host 192.168.1.252 and port not 123
• .252: udp and host 192.168.1.7 and port not 123

When I ran this, I saw plenty of TFTP traffic on a port other than 69. In this case, data packets went from port 60862 on the source host to 44792 on the destination, and acknowledgements came back the other way on these ports.

According to Wikipedia:

A transfer request is always initiated targeting port 69, but the data transfer ports are chosen independently by the sender and receiver during the transfer initialization. The ports are chosen at random according to the parameters of the networking stack, typically from the range of ephemeral ports.

If I understand this correctly, I will need to tunnel more than just port 69. What additional command(s) do I need to get these ports forwarded? Is there an easy way to get this done?

Update 5 Aug: I might have another problem before this.

Regarding the item listed above, I tried playing around with TFTP a little bit more and watching the packets in Wireshark. It looks like I get the first packet on port 69 from a random source port. In the last case, this was port 48723.

It turns out the TFTP server sends UDP packets back over this port from another port. If I can monitor the first packet to port 69, I can see this and I'm expecting that I can quickly set up another tunnel similarly on this port before it times out, so the process will be a bit manual, but I might be able to pull it off.

Now that I look closely, it looks like I'm not getting that first packet at all over socat.

When I have no tunnel, I'm seeing the initial packet on port 69 with no problem.

When I instead set up a direct socat with no SSH tunnel as follows, I see no packet.

Window 1:

$ sudo socat -d -d -d -d udp4-recvfrom:69,reuseaddr,fork udp:192.168.1.252:69

Window 2:

$ tftp localhost
tftp> get pxelinux.0
Transfer timed out.

Here is the output from netcat:

$ netstat -an | grep 69 | grep udp
udp        0      0 0.0.0.0:69              0.0.0.0:*

I also have one Wireshark instance running on both sides; both show absolutely nothing. Here is my capture filter on both sides:

udp and not (port 123 or port 5353 or port 1900)

I opened another question about this.

Update: A small step forward. I suspect I have something wrong with my socat command now.

I tried another simple test on my laptop only.

Window 1:

$ sudo nc -v -l -u 69

Wireshark:

• Listen on interface lo
• Capture filter: udp and port 69

Window 2:

$ nc -u 127.0.0.1 69

As I typed in Window 2, I saw the output in Window 1 and I saw packets captured in Wireshark.

Then, I repeated with my laptop's network card:

Window 1:

$ sudo nc -v -l -u 69

Wireshark:

• Listen on interface wlan0
• Capture filter: udp and port 69

Window 2:

$ nc -u 127.0.0.1 69

It turns out I wasn't getting any packets captured in Wireshark although I was getting the messages show up in Window 1. I remembered I set up Docker a while back so I had a docker0 bridge; I also have a couple virtualization products, so maybe they were getting in the way.

I re-ran the Wireshark capture with:

• Listen on interface any
• Capture filter: udp and port 69

This time, I was getting everything I wanted. Maybe my problem was the Wireshark capture interface all along.

Stopped the netcat windows, and taking one step back now:

$ sudo socat -d -d -d -d udp4-recvfrom:69,reuseaddr,fork udp:192.168.1.252:69

Tried the TFTP test again:

$ tftp localhost
tftp> get pxelinux.0

That was it! I'm getting my packets captured in Wireshark now!

I see the Read Request messages captured, and it turns out all five tries/retries come from the same source port.

Now time to see if I can manually set up an additional socat session on this source port. On the TFTP server itself, I additionally ran:

$ for f in 52163; do socat -d -d -d -d udp4-recvfrom:${p},reuseaddr,fork udp:192.168.1.7:${p}; done

Note this had the IP address of my laptop. (The for loop is simply for convenience so I only need to specify the port number once in the command.)

It turns out each of the source ports is the same no matter how many times I run the PXE get pxelinux.0 command; the source port number changes when I exit TFTP and re-run TFTP.

I have Wireshark running on both my laptop and the TFTP box; I'm now seeing packets on my laptop's Wireshark but not on the TFTP Wireshark.

I guess there is something wrong with my socat command, but I don't know what it is:

$ sudo socat -d -d -d -d udp4-recvfrom:69,reuseaddr,fork udp:192.168.1.252:69

Update: So now that I have the basics down for netcat, I thought I'd take a step up by introducing socat into the mix but still no ssh tunnel.

I have my local laptop, and for now let's call the TFTP server the remote box even though it's just a VM on my laptop.

Window 1:

local$ sudo socat udp4-recvfrom:69,reuseaddr,fork udp:192.168.1.13:69

Window 2:

local$ nc -4u localhost 69

Wireshark running on remote box capturing UDP traffic.

In Window 2, I typed three lines, pressing Enter after each one: "one", "two", and "three".

In Wireshark, I see three packets on the TFTP port, so the TFTP server must be getting this garbage data. What's interesting here is each of the three packets has a different source port.

Next, I exited netcat and instead ran the TFTP client specifying the local IP address instead of "localhost" since I got burned by that today:

local$ tftp 127.0.0.1
tftp> get pxelinux.0

Now I'm getting the TFTP Read Request packet from my local laptop!!

(FYI prior to capturing my notes here, I ran "tftp localhost" and yet again got nothing. It looks like the IPv4/IPv6 problem mentioned in my other question.)

So now back to the SSH tunnel.

Window 1:

remote$ socat tcp4-listen:6901,reuseaddr,fork udp:127.0.0.1:69

Window 2:

local$ ssh -L 6901:127.0.0.1:6901 192.168.1.13

Window 3:

local$ sudo socat udp4-recvfrom:69,reuseaddr,fork tcp:127.0.0.1:6901

I'm getting the Read Request packets captured in Wireshark along with the first Data Packet that can't get anywhere since there's no tunnel back over the source port, but at least I'm getting the UDP packets over the tunnel to the TFTP server!

Now this had me thinking about that return trip a bit. With socat in the middle, it looks like I'm getting five transfer attempts from this TFTP client before it times out; I don't know how the TFTP client in the NIC firmware will behave.

For each of these attempts, it seems I'm getting a different source port now since the connection to the TFTP server is coming from socat and not my laptop's TFTP client. Again, I have no idea how the TFTP client built into the NIC firmware will behave.

Even if I do monitor the traffic and set up a return tunnel fast enough after seeing the traffic in Wireshark, I'm not sure if that's too late.

I'm guessing the same will be true if I used pipes instead of socat.

Update: I think I'm ready to give up on this, but just for fun, I thought I'd go back to the original box and try to capture some packets now that I have enough to at least get one trip going.

My notion of "local" and "remote" is now reversed as my TFTP server is local to me whereas my dead PC is remote.

Window 1:

laptop$ ssh user@192.168.1.13
local$ socat tcp4-listen:6901,reuseaddr,fork udp:127.0.0.1:69

Window 2:

laptop$ ssh -R 6901:127.0.0.1:6901 user@remote.fqdn.net
remote$ ping -i 60 8.8.8.8

(The ping is to keep the connection alive in case it's needed.)

Window 3:

laptop$ ssh user@remote.fqdn.net
remote$ sudo socat udp4-recvfrom:69,reuseaddr,fork tcp:127.0.0.1:6901

I've been worried about the source port changing with socat, so I was curious to try the pipes again. I went back to my test setup.

I'm keeping my local and remote reversed for consistency with the production environment even though it's backwards for my test environment.

Window 1:

laptop$ ssh 192.168.1.13
local$ mkfifo /tmp/fifo
local$ nc -lp 6901 < /tmp/fifo | nc -u 127.0.0.1 69 > /tmp/fifo

Window 2:

laptop$ ssh 192.168.1.13
local$ ssh -R 6902:127.0.0.1:6901 192.168.1.7
remote$ ping -i 60 8.8.8.8

Window 3:

remote$ mkfifo /tmp/fifo
remote$ sudo nc -lup 69 < /tmp/fifo | nc 127.0.0.1 6902 > /tmp/fifo

I tried a simple TFTP get from my TFTP client again. This time, I noticed in Wireshark that the source port is the same for each packet! Maybe there is still hope!!

Remember the caveat for this is that the UDP packets stay below the size of the MTU, which I believe I read somewhere that they will be for TFTP.

I have to clean up my pipes:

local$ rm /tmp/fifo
remote$ rm /tmp/fifo

I wanted to try this on production now.

Window 1:

laptop$ ssh user@192.168.1.13
local$ mkfifo /tmp/fifo
local$ nc -lp 6901 < /tmp/fifo | nc -u 127.0.0.1 69 > /tmp/fifo

Window 2:

laptop$ ssh user@192.168.1.13
local$ ssh -R 6901:127.0.0.1:6901 user@remote.fqdn.net
remote$ ping -i 60 8.8.8.8

(The ping is to keep the connection alive in case it's needed.)

Window 3:

laptop$ ssh user@remote.fqdn.net
remote$ mkfifo /tmp/fifo
remote$ sudo -i
remote$ nc -lup 69 < /tmp/fifo | nc 127.0.0.1 6901 > /tmp/fifo

When I had the PC booted up, I got one TFTP packet come all the way through to the PXE server according to Wireshark, but I didn't get any of the retries. I was hoping to get the retries as well, and that all of them would have the same source port.

I changed the nc's on the left side of the pipes to also specify -k to keep alive, and then rebooted, but still only got the first packet.

I have to clean up my pipes:

local$ rm /tmp/fifo
remote$ sudo rm /tmp/fifo

$ ssh root@port.forwarded.pi -R 69:69

ssh -L 33066:localhost:3306 remotemachine

# tail /var/log/syslog