Ssh – Are remote LUKS volumes mounted locally with sshfs made visible to remote users

Your approach looks good. Some remarks though:

• If you want to encrypt rootfs, you'll need to use initrd (to have some minimal unencrypted system that will process the encrypted partitions).

  If the USB device is removable, both initrd and kernel can be stored on the USB to heighten tamper resistance (supposing you make sure the USB won't get into unauthorized hands) - which is usually why one encrypts rootfs. Once both kernel and initrd are on a removable media, you can ensure that nobody changes the kernel (or initrd) from the running system by simply removing the media.

  This is of course not an option if you want to have it inside of a server, but then again the question stands whether it makes sense to have such a device at all and not to use a small partition on one of the hard-drives. If for example all drives in the machine are in RAID, one would probably want to put rootfs on the USB as well. An interesting alternative to an internally connected USB flash device could be a CompactFlash card attached to ATA interface through an adapter, by the way.

  Some distributions offer prepared solutions for encrypted root, some don't - but mostly it's question of putting a couple of lines into initrd script before it tries to mount the "real" root (see for example man pages for pivot_root, usually in sections 2 (syscall) and 8 (bonary), if you're not familiar with the process).

• remember to backup the keys and passphrases in case your USB drive dies. LUKS follows rather one-sided approach when it comes to damaging its header - once a single sector of its header (key-slot to be more precise) dies, you are unable to mount it. This is to make sure that erasing the header isn't effectively thwarted by block reallocation performed by the device itself (because that's what flash-memory based devices do a lot) - the key is spread over the whole key slot and one needs all of the data to reconstruct it - there is no redundancy. See Clemens Fruwirth's website for deeper discussion.

  That said, maybe a simple encrypted device on the USB would be enough (check section PLAIN MODE in man cryptsetup). Or a file encrypted with e.g. openssl enc. The former might actually be an option even for the encrypted partitions themselves.

I'd assume the weakest link here is the SSHFS code -- the rest of the stuff is in kernel and pretty heavily used, so it's probably fine. I've never actually looked at any FUSE code before, so there could be something else going on that I've missed, but according to the SSHFS source code, SSHFS's implementation of fsync() doesn't do a whole bunch, it just calls flush() on the IO stream.

static int sshfs_fsync(const char *path, int isdatasync,
                       struct fuse_file_info *fi)
{
    (void) isdatasync;
    return sshfs_flush(path, fi);
}

At sshfs.c:2551, we can see that sshfs_flush() function doesn't send any sort of sync command to the remote machine that enforces an fsync. I believe the sshfs.sync_write flag means "wait for commands to go to the server before returning from write", not "fsync on the server on every write" because that second meaning would be very odd. Thus your fsync measurement is slower because it's bottlenecked by network speed, not remote disk speed.

static int sshfs_flush(const char *path, struct fuse_file_info *fi)
{
    int err;
    struct sshfs_file *sf = get_sshfs_file(fi);
    struct list_head write_reqs;
    struct list_head *curr_list;

    if (!sshfs_file_is_conn(sf))
        return -EIO;

    if (sshfs.sync_write)
        return 0;

    (void) path;
    pthread_mutex_lock(&sshfs.lock);
    if (!list_empty(&sf->write_reqs)) {
        curr_list = sf->write_reqs.prev;
        list_del(&sf->write_reqs);
        list_init(&sf->write_reqs);
        list_add(&write_reqs, curr_list);
        while (!list_empty(&write_reqs))
            pthread_cond_wait(&sf->write_finished, &sshfs.lock);
    }
    err = sf->write_error;
    sf->write_error = 0;
    pthread_mutex_unlock(&sshfs.lock);
    return err;
}

Note that it's possible that the remote SFTP implementation does actually fsync on writes, but I think that's actually not what's happening. According to an old draft of the SFTP standard (which is the best I can find) there is a way to specify this behavior:

7.9. attrib-bits and attrib-bits-valid
...
SSH_FILEXFER_ATTR_FLAGS_SYNC
       When the file is modified, the changes are written synchronously
       to the disk.

which would imply that this isn't the default (as it's faster to not fsync). According to that standards document there doesn't appear to be a way to request a fsync on the remote file, but it looks like OpenSSH supports this as an extension to SFTP

/* SSH2_FXP_EXTENDED submessages */
struct sftp_handler extended_handlers[] = {
    ...
    { "fsync", "fsync@openssh.com", 0, process_extended_fsync, 1 },
    ...
};

static void
process_extended_fsync(u_int32_t id)
{
    int handle, fd, ret, status = SSH2_FX_OP_UNSUPPORTED;

    handle = get_handle();
    debug3("request %u: fsync (handle %u)", id, handle);
    verbose("fsync \"%s\"", handle_to_name(handle));
    if ((fd = handle_to_fd(handle)) < 0)
        status = SSH2_FX_NO_SUCH_FILE;
    else if (handle_is_ok(handle, HANDLE_FILE)) {
        ret = fsync(fd);
        status = (ret == -1) ? errno_to_portable(errno) : SSH2_FX_OK;
    }
    send_status(id, status);
}

I doubt it'd be hard to query for that extension and properly support fsync in SSHFS, that seems a pretty reasonable thing to do. That said, I think it'd probably be easier to just use Linux's network block device support which I assume supports all this stuff properly (though I've never used it myself, so it could be horrible).