I had the same problem, and I spent several days trying different .ISO files, USB sticks, UEFI and secure-boot settings, all to no avail.
I finally did find a workaround, but you're not going to like it because it is time-consuming. First install Ubuntu 16.04, and then upgrade to 18.04 with this:
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
The upgrade takes a few hours, but at least it works.
To your working ssh setup, add a vnc server like x11vnc to your server and a viewer like xtightvncviewer to your remote machine. For outside the local network, ensure you have a passthrough for your ssh (port 5900 will be tunneled, so nothing special needed for it).
X should be running on the server, but no existing login is needed (you will connect to the login screen). No other special services needed on the server.
From the remote, use ssh to start the x11vnc server, and tunnel port 5900 back to the remote (assumed to be on ...131 on the local network). In a terminal type:
ssh -t -L 5900:localhost:5900 192.168.1.131 "sudo x11vnc -localhost -create -auth guess -display :0"
The output should indicate a successful connection. Leave this terminal and in another, start the vnc viewer on the (tunneled local port 5900):
xtightvncviewer -encodings RRE -bgr233 localhost::5900
The encodings and option improve performance for me, but are optional. A new window should open with the server login screen displayed. Login to the desktop you set up.
If you want to leave the server program running over multiple invocations of the viewer, you may use the -forever option, but with the above, the server will close the x11vnc program when the viewer exits.
Best Answer
I suggest that you start from an old style server iso file (with the Debian installer), that you find at
cdimages.ubuntu.com/ubuntu/releases/bionic/release/
Install a minimal system (without any extry program package).
Boot into the minimal system and install Ubuntu desktop
After rebooting you should have Ubuntu. I don't know what you mean by a minimal installation, and I don't know why you want to start from Ubuntu Server.
You can install a simple window manager instead of the Ubuntu desktop to get a very light graphical system, that you can start with
startx,or you try with
which might produce what you want.
You get other light-weight desktop environments via