Dnsmasq not working oh Mac OS High Sierra

mDNSResponder and dnsmasq have to run both: dnsmasq is the lightweight DNS-server (and DHCP/Router) and mDNSResponder is responsible for all local queries.

To set up dnsmasq in OS X in your environment properly do the following:

Remove any DNS-server in the network preferences of the dnsmasq host (your MacBook Air) except 127.0.0.1.

Remove any DNS-server in the network preferences of the VMs in use and replace them by the IP-address of the VM-host (your MacBook Air).

Remove any file in /etc/resolver/. Usually they aren't necessary. You may keep them but then they probably should have this form:

/etc/resolver/rab with the content

nameserver    127.0.0.1

/etc/resolver/tig with the content

nameserver    127.0.0.1

The logic behind this is mentioned in resolver(5):

domain
    Domain name associated with this resolver configuration. This option is normally not     required by the Mac OS X DNS search system when the resolver configuration is read from a     file in the /etc/resolver directory. In that case the file name is used as the domain name.     However, domain must be provided when there are multiple resolver clients for the same     domain name, since multiple files may not exist having the same name. See the SEARCH     STRATEGY section for more details.

Then edit /usr/local/etc/dnsmasq.conf and add/modify

server=/rab/192.168.0.2 #your main IP-address or 127.0.0.1
server=/tig/192.168.0.2 #your main IP-address or 127.0.0.1
server=8.8.8.8 #forwarder
addn-hosts=/usr/local/etc/hosts/hosts.conf

Now add the hosts.conf file in /usr/local/etc/hosts/ with the content:

127.0.0.1   localhost
192.168.56.1    abc.rab
192.168.56.2    abc.tig

Then restart dnsmasq with launchctl to load the new conf files.

Since your Windows VM already is in the bridged Wi-Fi network you can completely dump the host-only network and modify /usr/local/etc/hosts/hosts.conf:

127.0.0.1   localhost
192.168.0.2 abc.rab #IP-address of the MacBook Air Wi-Fi interface
192.168.0.3 abc.tig #IP-address of the Windows bridge interface

Your dnsmasq daemon isn't properly configured.

Your external resolver is working: all queries to non-dev hosts/domains are forwarded to 3rd-party DNS servers with the resolv-file=/usr/local/etc/resolv-dnsmasq.conf line - in your case the configured file contains two public Google DNS servers.

Your internal resolver doesn't resolve internal names though.

The line address=/.dev/127.0.0.1 or better address=/dev/127.0.0.1 will redirect any *.dev query to the host 127.0.0.1. An internal resolver is not needed then and the internal name server defined in /etc/resolver/dev is useless.

Compare this with the example in the dnsmasq.conf file:

# Add domains which you want to force to an IP address here.
# The example below send any host in double-click.net to a local
# web-server.
#address=/double-click.net/127.0.0.1

Any query for *.double-click.net will be redirected to 127.0.0.1 and to an arbitrary website served at localhost.

I strongly recommend to define a hosts.config file and enter/define all necessary hosts there:

Add a line addn-hosts=/usr/local/etc/hosts/hosts.conf in dnsmasq.conf. Then add a folder with sudo mkdir /usr/local/etc/hosts and create a file hosts.conf

sudo nano /usr/local/etc/hosts/hosts.conf

with the following content:

127.0.0.1   localhost
127.0.0.1   test.dev
127.0.0.1   test2.dev
...

After saving the file reload your dnsmasq daemon.

If you want to use different IPs for your host names e.g.:

127.0.0.1   localhost
127.0.0.2   test.dev
127.0.0.3   test2.dev
...

you'd have to add additional IPs with:

sudo ifconfig lo0 alias 127.0.0.2 up
sudo ifconfig lo0 alias 127.0.0.3 up
...