Systems not seeing netrestore image

firewallNetworkosx-server

Turning to here after many futile attempts to resolve.

I have an OSX server (10.10.5) which has been hosting a Netrestore image over HTTP without issue for some time now. We replaced our firewall in our network with one from another vendor, which has the exact same network configuration (sans VLANs, though that is a moot point. Read below) and yet since the swap, the Macbooks in our organization cannot see the hosted images. Here is what I am facing:

  • Systems on same VLAN as the servers (normally not, but doesn't work either way.)
  • Firewall not enabled on OSX server
  • All required ports (both UDP and TCP) are reachable on the server
  • Images are NOT limited to model, and existing machines with known working configurations against the images cannot see them either
  • Netinstall service has been restarted, images have not changed, services show online
  • Tried delivering images both over HTTP and NFS
  • Server is pingable by all machines
  • TCP dump for bootpd does not show anything hitting the server
  • Server switched from manual IP config to DHCP but retains same IP, Sub and Gateway

The systems which are trying to reach the server exhibit behavior as if the server is not delivering the image. I'm not sure where to go for the next step at this point, or where to look next. Any thoughts?

Best Answer

It sounds like something in the network is blocking nonstandard DHCP traffic, including the requests a NetBoot client uses to find the server(s). When you start a Mac with the Option or N keys held down, it'll first do a normal DHCP transaction to get an IP address, then send a special "BSDP" (Boot Service Discovery Protocol) request, which is really a DHCP Inform request with some special options set. Here's what it looks like with sudo tcpdump -nv -s0 port bootps:

16:19:35.656369 IP (tos 0x0, ttl 64, id 10411, offset 0, flags [DF], proto UDP (17), length 328)
    10.0.0.215.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 68:5b:35:xx:xx:xx, length 300, Flags [none]
      Client-IP 10.0.0.215
      Client-Ethernet-Address 68:5b:35:xx:xx:xx
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        Vendor-Option Option 43, length 7: 1.1.1.2.2.1.1
        DHCP-Message Option 53, length 1: Inform
        Parameter-Request Option 55, length 2: 
          Vendor-Option, Vendor-Class
        MSZ Option 57, length 2: 1500
        Vendor-Class Option 60, length 28: "AAPLBSDPC/i386/MacBookPro9,2"
        Client-ID Option 61, length 7: ether 68:5b:35:xx:xx:xx

Note that the source address is a unicast address, not 0.0.0.0 like a normal DHCP request would have; your firewall may think that's hinky and block it. Or it might be blocking it for some other reason. Anyway, if the NetBoot server receives this, it should reply with something like this:

16:19:35.656756 IP (tos 0x0, ttl 64, id 59742, offset 0, flags [none], proto UDP (17), length 369, bad cksum 0 (->7b45)!)
    10.0.0.2.67 > 10.0.0.215.68: BOOTP/DHCP, Reply, length 341, Flags [none]
      Client-IP 10.0.0.215
      Client-Ethernet-Address 68:5b:35:xx:xx:xx
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: ACK
        Server-ID Option 54, length 4: 10.0.0.2
        Vendor-Class Option 60, length 9: "AAPLBSDPC"
        Vendor-Option Option 43, length 78: 1.1.1.4.2.127.255. [trimmed...]
16:19:35.657252 IP (tos 0x0, ttl 64, id 33486, offset 0, flags [none], proto UDP (17), length 328, bad cksum 0 (->e1fe)!)
    10.0.0.2.67 > 10.0.0.215.68: BOOTP/DHCP, Reply, length 300, Flags [none]
      Client-IP 10.0.0.215
      Server-IP 10.0.0.2
      Client-Ethernet-Address 68:5b:35:xx:xx:xx
      sname "mainserver.pretendco.com"
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: ACK
        Server-ID Option 54, length 4: 10.0.0.2

...and if the client receives that, it'd use the information in it to fetch the booter and kernel over TFTP, and then mount the image over either HTTP or NFS and actually boot from it.

BTW, this tends to be a pain to troubleshoot because you have to boot the client over & over to get it to send the requests. But there's a trick: boot a client normally, then open System Preferences -> Startup Disk, and it'll scan for NetBoot images using the same BSDP/DHCP queries. Much easier, plus you can run packet captures on the client as you do it. Only tricky thing is that to make it rescan, you have to fully quit System Preferences, not just leave & reenter the Startup Disk pane.

Related Question