You made some mistakes, but the main problem is in ubiquity and grub. Basically, when you set / to be an encrypted partition and don't create a separate partition for /boot, grub gives an error message like:

I know /boot is encrypted. You need to set GRUB_ENABLE_CRYPTODISK=y in /etc/default/grub. I won't do it for you, so I'm going to fail and your installation will stop.

An overview of the process

• We use EFI mode.
• We install to an unencrypted /boot partition and an encrypted btrfs / using the standard installer.
• After the installer finishes, we chroot, make some important configuration changes, and re-install grub to the EFI System Partition and re-create initrd.

The detailed steps

• Boot from Ubuntu 16.04 install disk (tested with Xubuntu).
• Connect to the Internet and run sudo apt update && sudo apt upgrade to update the installer components
• Use fdisk, gparted, or another tool to create 3 partitions:
  • A GPT partition table
  • A 200MB partition that we will use for the EFI System Partition
  • A multi-gigabyte partition that we will eventually use as our encrypted swap partition, but which will function as our temporary unencrypted /boot
  • An encrypted partition that uses the rest of the space

• Prepare the encrypted partition

  sudo cryptsetup luksFormat /dev/sda3
  sudo cryptsetup luksOpen --allow-discards /dev/sda3 sda3_crypt
  sudo mkfs.btrfs /dev/mapper/sda3_crypt
  

• Install Ubuntu

  • Choose "Something else" when asked about installation type.
  • Configure /dev/sda1 as EFI System Partition
  • Configure /dev/sda2 as ext2, formatted, with mount point of /boot
  • Configure /dev/mapper/sda3_crypt as btrfs with mount point of /
  • Continue with the installation.
  • After it finishes, choose to stay in the live system (no reboot).

• Copy the contents of /boot and do a chroot

  sudo mount -o subvol=@ /dev/mapper/sda3_crypt /target
  sudo mount /dev/sda2 /mnt
  # (Watch those trailing slashes! rsync is very sensitive to them.)
  sudo rsync -aXAH /mnt/ /target/boot/
  sudo mount /dev/sda1 /target/boot/efi
  sudo mount --bind /dev /target/dev
  sudo mount --bind /proc /target/proc
  sudo mount --bind /sys /target/sys
  sudo chroot /target
  

• (Everything is now happening as chroot inside your new system.)

• Add line to /etc/default/grub

  GRUB_ENABLE_CRYPTODISK=y
  

• Add line to /etc/crypttab. You will need to first run sudo blkid to find the UUID of /dev/sda3 (NOT /dev/mapper/sda3_crypt).

  sda3_crypt UUID=<UUID of /dev/sda3> none luks,discard
  

• Edit /etc/fstab and delete the line for /boot. The other entries are correct.

• Install grub to the EFI System Partition, generate a new grub.cfg, and prepare initrd.

  sudo grub-install --target=x86_64-efi --efi-directory /boot/efi --bootloader=ubuntu --boot-directory=/boot/efi/EFI/ubuntu --recheck
  sudo grub-mkconfig -o /boot/efi/EFI/ubuntu/grub/grub.cfg
  sudo update-initramfs -c -k all
  

• Optional double-check: Double-check that /boot/efi/EFI/ubuntu/grub/grub.cfg contains lines that include insmod luks, cryptomount -u <UUID>, the correct boot entries, etc. And double-check that your initrd contains the cryptsetup binary. If these things are missing, it is because grub-mkconfig and/or update-initrd couldn't figure out how the volumes that you've mounted or specified in fstab relate to the encrypted volume in crypttab. (There's a lot of magic autoconfiguration that they do.) This may happen if you diverge from this guide by, for example, using ZFS or by trying to partition sda3_crypt.

• (If using ZFS instead of btrfs) grub-mkconfig and update-initrd won't recognize ZFS. The workaround involves (during chroot, prior to grub-mkconfig/update-initrd) editing /usr/sbin/grub-mkconfig to add || true to line 139 (which starts with GRUB_DEVICE=), adding GRUB_DEVICE="/dev/mapper/sda3_crypt" to /etc/default/grub, creating file /usr/share/initramfs-tools/conf-hooks.d/forcecryptsetup with contents export CRYPTSETUP=y and file /etc/initramfs-tools/conf.d/cryptroot with contents target=sda3_crypt,source=UUID=<UUID of sda3>,key=none,discard. All of this is in addition to steps that you would take if you were not encrypting the ZFS partiion (like installing zfs userspace utils in both the live system and during chroot and deleting the line that mounts / in fstab).

• Exit chroot and reboot into your new system

  exit
  sudo umount /target/boot/efi
  sudo umount /target/dev
  sudo umount /target/proc
  sudo umount /target/sys
  sudo umount /target
  sudo reboot
  

• You should see grub asking for your password. Then you'll get the boot menu. After choosing Ubuntu you'll be asked for your password again. Then you'll be in your system. Read more about how Ubuntu uses BTRFS.

• TODO: Create encrypted swap (hint: it involves editing crypttab, fstab, and re-running update-initrd).

• TODO: Save your password so you only need to enter it once into grub. This is detailed here.

Upgrades

• Every time you install a new kernel, you should run the custom grub-mkconfig command.
• Every time you update grub, you should run the custom grub-installcommand.

Other notes

• It's tempting to create a single encrypted volume and partition it to create the swap partition (and possibly others), but this does not work. Both grub-mkconfig and update-initrd will misbehave. However, I haven't tried LVM.
• It may be tempting to use a swapfile on top of btrfs, but it's probably a bad idea because of performance.