Ubuntu – How to get dropbear to actually work with initramfs

initramfsserver

I'm trying to unlock a headless server running an encrypted version of Ubuntu 14.04. It is a clean 14.04.2 install with all updates as of writing.

I went thtrough the standard shenanigans with dropbear and busybox, i.e.:

# INSTALL
sudo apt-get install dropbear busybox           # on server

# ENABLE AND CONFIGURE IP
sudo sed -i 's/NO_START=1/NO_START=0/g' /etc/default/dropber
sudo sed -i 's/BUSYBOX=y/BUSYBOX=y\nDROPBEAR=y\n/g' \
            /etc/iniramfs/iniramfs.conf
sudo sed -i 's/DEVICE=.+/DEVICE=etho0/g' \
             /etc/iniramfs/initramfs.conf
sudo sed -i 's/IP=.+/IP=IP=192.168.0.11:::255.255.255.0::eth1:off' \
            /etc/initramfs/initramfs.conf
sudo update-initramfs

# COPY DROPBEAR SSH KEY
# WRONG: sudo cp /etc/dropbear/dropbear_*_host_key /tmp
sudo cp /etc/initramfs-tools/root/.ssh/id_rsa /tmp             # BETTER!
sudo chown $USER:$USER /tmp/id_rsa

scp server:/tmp/id_rsa ~/.ssh/id_rsa_dropbear_server           # on client

sudo reboot                                                    # on server

# CONNECT TO SERVER
ssh -vv -i ~/.ssh/id_rsa_dropbear_server \
        -o 'UserKnownHostsFile=~/.ssh/known_hosts.initramfs' \ 
         root@192.168.0.11                                     # on client

So I copied the auto-generated ssh-key and did a minimal configuration of dropbear. To my surprise ~~two~~ three things are malfunctioning:

The system ignores the IP settings. I found in my router's "connected devices" site, that the server has the ip 192.168.0.27 despite my configuration. So I tried the listed wrong IP with: ssh -vv -i ~/.ssh/dropbear_dss_host_key -o 'UserKnownHostsFile=~/.ssh/known_hosts.initramfs' root@192.168.0.27. That connects to dropbear but:
Dropbear ignores all public keys other than its own in /etc/initramfs-tools/root/.ssh/authorized_keys (tested with dss – maybe also rsa).
Solved: ~~Dropbear wants a passphrase for the key, which I do not have. So I tried an empty passphrase. With which dropbear drops to password authentication and wants the root password, that is not set.~~
Custom hook scripts seem to be partially ignored. These are scripts that worked in older install!

Here is the whole ssh session.

I added my usual public key file to dropbear's known hosts in the server's /etc/initramfs-tools/root/.ssh/authorized_keys and tried to ssh with my usual key. That didn't work.

I added the line GRUB_CMDLINE_LINUX_DEFAULT="ip=192.168.0.11::192.168.0.1:255.255.255.0::eth0:none" to the server's /etc/default/grup and updated grup. That was meant to fix the IP issue. But that didn't work either.

I'm now thoroughly annoyed and at the end of my patience. Where did I go wrong? Also, is the syntax for the IP settings right because one guide says ::eth0:off and the next says ::etho:none?

Edit

There is someone with what seems to be the same problem on 15.04.

Edit 2

I can now connect to the server. Turns out, I had copied the wrong private key to use with dropbear. The error has been corrected in my script above. But adding keys still doesn't work (i.e. to dropbear's authorized_keys file). There's word you need to convert the public keys, that you want to add to /etc/initramfs-tools/root/.ssh/authorized_keys to dropbear's format, but I don't want to spend time searching how. I only tried dss public keys. Maybe dropbear just likes rsa better?

Also I noticed, that custom hook scripts don't seem to work. They are not included in the initramfs' directories but lsinitramfs -l /boot/initrd.img-3.16.0-43-generic lists them as part of the image. The IP settings are still ignored as well. Even if I add GRUB_CMDLINE_LINUX_DEFAULT="ip=192.168.0.11::192.168.0.1:255.255.255.0::eth0:none" to the grubconfig in /etc/default/grub and update everything.

Edit 3

So it seems that /usr/lib/dropbear/dropbearconvert INPUTFORMAT OUTPUTFORMAT INFILE OUTFILE is the program to convert keys. The FORMAT parameter can be either openssh or dropbear. But it doesn't seem that's the answer on how to add keys to the servers /etc/initramfs-tools/root/.ssh/authorized_keys. The existing key there already is in openssh's public key file format. So adding other openssh-format keys shouldn't be a problem. Yet it is.

Best Answer

It's unclear which known_hosts are troubling you - on the server or on the client. I used the instructions at http://blog.neutrino.es/2011/unlocking-a-luks-encrypted-root-partition-remotely-via-ssh/ on Debian. I am going to paraphrase that page and add some tweak I found useful. Contemporary Debian dropbox packages create all the needed keys on the server at installation time, but the referenced tutorial is so old, that it mentions cases when the keys need to be manually created. YMMV.

A. On the server. As you correctly mentioned, dropbear and OpenSSH keys differ, but are supposedly inter-convertible. The tutorial says that you could create on the server the host keys with:

dropbearkey -t rsa -f /etc/initramfs-tools/etc/dropbear/dropbear_rsa_host_key

In fact, it is a bit harder than that. For me on Debian Jessie, the steps were:

Create a OpenSSH key.

ssh-keygen -t rsa -b 4096 -m PEM -f bootkey_rsa
Copy it on the server, that is, the computer where you are more likely to have dropbearconvert. There, convert it to dropbear format:

/usr/lib/dropbear/dropbearconvert openssh dropbear bootkey_rsa bootkey_dropbear_rsa
Stil on the server, login to root, and extract from Dropbear key the public part to the file where boot dropbear looks for it:

dropbearkey -y -f bootkey_dropbear_rsa | grep "^ssh-rsa" > /etc/dropbear-initramfs/authorized_keys
Update initramfs:

update-initramfs -u -k all
Cleanup: remove bootkey_dropbear_rsa and bootkey_rsa from server.

Note that the location where the known_hosts is expected by update-initramfs, given above as /etc/dropbear-initramfs/authorized_keys , is flexible, and probably changes from distro to distro. To be sure you put it in the right place, read the source file from /usr/share/initramfs-tools/hooks/dropbear.

The other needed key is created with:

dropbearkey -t rsa -f /etc/initramfs-tools/root/.ssh/id_rsa.dropbear

B. On the client you need both kinds of keys, so there is another known_hosts. In the referenced tutorial, the command to connect is:

ssh -o "UserKnownHostsFile=~/.ssh/known_hosts.initramfs" \
-i "~/id_rsa.initramfs" root@my.server.ip.addr \
"echo -ne \"MyS3cr3tK3y\" >/lib/cryptsetup/passfifo"

One of the keys, id_rsa.initramfs, is the file /etc/initramfs-tools/etc/dropbear/dropbear_rsa_host_key or bootkey_rsa from the server section.

On the client known_hosts, there may be a conflict between the key you already have for the OpenSSH server and the dropbear server you just installed. I temporarily removed from known_hosts the keys for the regular OpenSSH service, connected using the command above (leaving out the -o parameter), got prompted if I trust the host-key, said yes, and got it appended to my known hosts. From there on, you need to move that last line to its own known_hosts (~/.ssh/known_hosts.initramfs in the example above).

Related Solutions

Ubuntu – Unable to ssh remote unlock encrypted ubuntu server 15.04 using dropbear/initramfs

I'm really not sure what the problem was; I tried using a different known-working dropbear id_rsa key and that still didn't work. I ended up completely reinstalling a fresh 15.04, and following all the steps again and was able to remote unlock without issue.

Ubuntu – Ubuntu will not boot into busybox for remote LUKS decryption using dropbear

After what feels like an eternity of deep diving into Google and trial and error, I finally got this figured out.

Here are the steps I took relative to the steps I outlined in the question:

Removed the script in the blogpost referenced in the question
In the crossfire of trial and error, ended up removing ifconfig eth0 0.0.0.0 down from usr/share/initramfs-tools/scripts/init-bottom/dropbear that step 6 in the blogpost outlined; I never added it back but never needed it

Modified and added the following scripts from this post:

# Comment lines in /usr/share/initramfs-tools/scripts/local-top/cryptroot as follows:
#
if [ -z "$cryptkeyscript" ]; then
cryptkey="Unlocking the disk $cryptsource ($crypttarget)\nEnter passphrase: "
#if [ -x /bin/plymouth ] && plymouth --ping; then
# cryptkeyscript="plymouth ask-for-password --prompt"
# cryptkey=$(echo -e "$cryptkey")
#else
cryptkeyscript="/lib/cryptsetup/askpass"
#fi
fi



# Add /usr/share/initramfs-tools/hooks/cryptroot_unlock and make executable
#
# Prompt to unlock LUKS encrypted root partition remotely
#
# See linked post for sources and acknowledgements
#
#!/bin/sh
PREREQ=""
prereqs()
{
echo "$PREREQ"
}
case $1 in
prereqs)
prereqs
exit 0
;;
esac
. /usr/share/initramfs-tools/hook-functions
#
# Begin real processing
#
SCRIPTNAME=unlock
# 1) Create script to unlock luks partitions
cat > ${DESTDIR}/bin/${SCRIPTNAME} << '__EOF'
#!/bin/sh
/lib/cryptsetup/askpass "Enter volume password: " > /lib/cryptsetup/passfifo
__EOF
chmod 700 ${DESTDIR}/bin/${SCRIPTNAME}
# 2) Enhance Message Of The Day (MOTD) with info how to unlock luks partition
cat >> ${DESTDIR}/etc/motd << '__EOF'
To unlock root-partition run "${SCRIPTNAME}"
__EOF



# Add /usr/share/initramfs-tools/scripts/local-bottom/dropbear_kill_clients and make executable
#
#
# Kills all DropBear client sessions if InitRAMFS is left
#
# See linked post for sources and acknowledgements
#
#!/bin/sh
PREREQ=""
prereqs()
{
echo "$PREREQ"
}
case $1 in
prereqs)
prereqs
exit 0
;;
esac
#
# Begin real processing
#
NAME=dropbear
PROG=/sbin/dropbear
# get all server pids that should be ignored
ignore=""
for server in `cat /var/run/${NAME}*.pid`
do
ignore="${ignore} ${server}"
done
# get all running pids and kill client connections
for pid in `pidof "${NAME}"`
do
# check if correct program, otherwise process next pid
grep -F -q -e "${PROG}" "/proc/${pid}/cmdline" || {
continue
}
# check if pid should be ignored (servers)
skip=0
for server in ${ignore}
do
if [ "${pid}" == "${server}" ]
then
skip=1
break
fi
done
[ "${skip}" -ne 0 ] && continue
# kill process
echo "$0: Killing ${pid}..."
kill -KILL ${pid}
done

After modifying and adding those scripts, dropbear was able to start, but my network device was failing to connect to the network, so I was still unable to connect to the server.

I finally discovered by using ls /sys/class/net that my network adapter was not called eth0; apparently that is an old notation no longer used by recent versions of Ubuntu, and since all the posts I have found are old if not ancient, eth0 is all I found references to.

So, armed with that information and a few more snippets I found from other sources, I modified initramfs.conf as follows:

Modified the DEVICE= portion of /etc/initramfs-tools/initramfs.conf to read:

DEVICE=<name of network adapter discovered using ls /sys/class/net>
IP=<Static IP Address>::<Default Gateway>:<Subnet Mask>::<name of network adapter>:off

Updated initramfs (sudo update-initramfs -u)

Now dropbear connects to the network and I can connect to the server and unlock remotely.