MacOS Network – How to Unblock a Port on MacOS

You can use the following command in the Terminal:

pfctl -s all

It will display a list of all sets of filter rules, translation rules, limits, etc. set on the network packet filter.

It sounds like something in the network is blocking nonstandard DHCP traffic, including the requests a NetBoot client uses to find the server(s). When you start a Mac with the Option or N keys held down, it'll first do a normal DHCP transaction to get an IP address, then send a special "BSDP" (Boot Service Discovery Protocol) request, which is really a DHCP Inform request with some special options set. Here's what it looks like with sudo tcpdump -nv -s0 port bootps:

16:19:35.656369 IP (tos 0x0, ttl 64, id 10411, offset 0, flags [DF], proto UDP (17), length 328)
    10.0.0.215.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 68:5b:35:xx:xx:xx, length 300, Flags [none]
      Client-IP 10.0.0.215
      Client-Ethernet-Address 68:5b:35:xx:xx:xx
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        Vendor-Option Option 43, length 7: 1.1.1.2.2.1.1
        DHCP-Message Option 53, length 1: Inform
        Parameter-Request Option 55, length 2: 
          Vendor-Option, Vendor-Class
        MSZ Option 57, length 2: 1500
        Vendor-Class Option 60, length 28: "AAPLBSDPC/i386/MacBookPro9,2"
        Client-ID Option 61, length 7: ether 68:5b:35:xx:xx:xx

Note that the source address is a unicast address, not 0.0.0.0 like a normal DHCP request would have; your firewall may think that's hinky and block it. Or it might be blocking it for some other reason. Anyway, if the NetBoot server receives this, it should reply with something like this:

16:19:35.656756 IP (tos 0x0, ttl 64, id 59742, offset 0, flags [none], proto UDP (17), length 369, bad cksum 0 (->7b45)!)
    10.0.0.2.67 > 10.0.0.215.68: BOOTP/DHCP, Reply, length 341, Flags [none]
      Client-IP 10.0.0.215
      Client-Ethernet-Address 68:5b:35:xx:xx:xx
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: ACK
        Server-ID Option 54, length 4: 10.0.0.2
        Vendor-Class Option 60, length 9: "AAPLBSDPC"
        Vendor-Option Option 43, length 78: 1.1.1.4.2.127.255. [trimmed...]
16:19:35.657252 IP (tos 0x0, ttl 64, id 33486, offset 0, flags [none], proto UDP (17), length 328, bad cksum 0 (->e1fe)!)
    10.0.0.2.67 > 10.0.0.215.68: BOOTP/DHCP, Reply, length 300, Flags [none]
      Client-IP 10.0.0.215
      Server-IP 10.0.0.2
      Client-Ethernet-Address 68:5b:35:xx:xx:xx
      sname "mainserver.pretendco.com"
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: ACK
        Server-ID Option 54, length 4: 10.0.0.2

...and if the client receives that, it'd use the information in it to fetch the booter and kernel over TFTP, and then mount the image over either HTTP or NFS and actually boot from it.

BTW, this tends to be a pain to troubleshoot because you have to boot the client over & over to get it to send the requests. But there's a trick: boot a client normally, then open System Preferences -> Startup Disk, and it'll scan for NetBoot images using the same BSDP/DHCP queries. Much easier, plus you can run packet captures on the client as you do it. Only tricky thing is that to make it rescan, you have to fully quit System Preferences, not just leave & reenter the Startup Disk pane.