Try disabling the display in the display settings, then open sound options and set HDMI as default device.
From our comments, It seems that the GPU is responsible for the signal that is sent to display, therefore it does not seem possible to only get audio from a GPU's output.
You don't configure PulseAudio to use the ALSA default device. Instead, you configure ALSA to use pulse as the default device:
pcm.!default pulse
ctl.!default pulse
In that way, ALSA applications that are not aware of PulseAudio will use PulseAudio via that indirection layer.
The reason you need to do this is that PulseAudio always uses ALSA as backend, and on startup opens all ALSA devices, and since most ALSA devices can't be opened multiple times, this will cause all ALSA applications that try to use an ALSA device directly when PulseAudio is running to fail.
If you have a legacy application that for some reason doesn't work, you can use pasuspender to temporary suspend PulseAudio to run this particular application.
I was actually in a similar situation like you are now, I used ALSA for a long time, and was happy, and was then forced to switch to PulseAudio. PulseAudio has in principle a nicer, more general structure than ALSA (streams and sources/sinks), and though sometimes it has warts, one can live with it.
You can debug problems by starting pulseaudio manually and adding -v flags, like pulseaudio -vv start etc. Log messages go to the syslog. If you want to switch devices, pavucontrol is an easy-to-use GUI, and if you want more direct control, you can use pacmd or pactl (I never figured out why there are two programs) as a commandline CLI. Use the help argument for more details. If you prefer files, everything easily scriptable.
I also recommend to have a look at the available modules.
Best Answer
As long as the device reports itself as a USB Audio device the ALSA driver will support it, although the exact definition of "support" may vary for more complex devices.