(CentOS 7, Ceph Incunabulus, everything patched up to date)

Background

I've got a working converged Ceph/qemu/kvm cluster with cephx enabled for authentication. I'm trying to use this to store block devices for qemu/kvm virtual machines – not an uncommon configuration. I'd like to set up a pool for virtual disks in ceph, define that pool and authenticate to it in libvirt, and create virtual machines using the pool. All the tutorials and examples seem to show setting up cephx authentication directly on each disk file. Given my day so far, I'm beginning to see why!

Question

When I try the below, qemu-kvm appears to get passed a drive configured as file=rbd:vmlive/testguest-vda:auth_supported=none. That should read auth_supported=cephx, and for some reason it ain't happening!

Can I put all the authentication in the pool as I want, or do I have to put an authentication section in each virtual disk's XML? If I can do this, where's my mistake in what's below?

What I'm trying now

Context

I have two nodes: test1 and test2. Each has two OSDs, each is also a monitor (yes, I'm aware running with an even number of monitors is a bad idea). Each has libvirt installed, hence qemu/kvm. I'm running the commands below as root on test1. The UUID has been changed, and you'll see I don't make any keys visible below!

Create a Ceph pool and a client.kvm user

SENSIBLE_PGS=512 # Calculate as OSDs * 100 / number of replicas, rounded up to next power of 2
ID=kvm
POOL=vmlive
ceph osd pool create $POOL $SENSIBLE_PGS $SENSIBLE_PGS
KEY=`ceph auth get-or-create client.$ID | grep -v '^\[' | awk '{print $3}'`
ceph auth caps client.$ID mon "allow r" osd "allow rwx pool=$POOL"

Create a libvirt secret holding the Ceph user's key

SECRET_DEFINITION_FILE=~/secret-$ID.xml
SECRET_UUID=12345678-1234-1234-1234-123456789abc
cat > $SECRET_DEFINITION_FILE << EOF
<secret ephemeral='no' private='no'>
  <uuid>$SECRET_UUID</uuid>
  <usage type='ceph'>
    <name>ceph client.$ID key</name>
  </usage>
</secret>
EOF
virsh secret-define --file $SECRET_DEFINITION_FILE
rm -f $SECRET_DEFINITION_FILE
virsh secret-set-value "$SECRET_UUID" "$KEY"

Create a libvirt storage pool, trying to use cephx authentication

POOL_DEFINITION_FILE=~/pool-$POOL.xml
cat > $POOL_DEFINITION_FILE << EOF
<pool type="rbd">
  <name>$POOL</name>
  <source>
    <name>$POOL</name>
    <host name="test1" port="6789" />
    <host name="test2" port="6789" />
    <auth username='$ID' type='ceph'>
      <secret uuid='$SECRET_UUID'/>
    </auth>
  </source>
</pool>
EOF
virsh pool-define $POOL_DEFINITION_FILE
rm -f $POOL_DEFINITION_FILE
virsh pool-autostart $POOL
virsh pool-start $POOL

Create a guest using virt-install

NAME=testguest
DEV=vda
SIZE=8G
IMAGE=CentOS-7-x86_64-Minimal-1503-01.iso
VCPUS=1
RAM=512
MACLAST_HEX=10
IPLAST_DECIMAL=16

FILE=$NAME-$DEV
qemu-img create -f rbd rbd:$POOL/$FILE $SIZE
sudo virsh net-update default add-last ip-dhcp-host \
  --xml "<host mac='52:54:00:00:00:$MACLAST_HEX' ip='192.168.122.$IPLAST_DECIMAL' name='$NAME'/>" --live --config
sudo virt-install \
  --connect qemu:///system \
  --virt-type kvm \
  --name $NAME \
  --ram $RAM \
  --vcpus=$VCPUS \
  --disk vol=$POOL/$FILE \
  --location /var/lib/libvirt/images/$IMAGE \
  --vnc \
  --noautoconsole \
  --os-type linux \
  --os-variant rhel7 \
  --network=bridge:virbr0,model=virtio,mac=52:54:00:00:00:$MACLAST_HEX \
  --autostart

The result

WARNING  vmlive/testguest-vda may not be accessible by the hypervisor. You will need to grant the 'qemu' user search permissions for the following directories: ['vmlive', '']

Starting install...
Retrieving file .treeinfo...
Retrieving file vmlinuz...
Retrieving file initrd.img...
ERROR    internal error: process exited while connecting to monitor: 2016-01-06T15:16:54.639890Z qemu-kvm: -drive file=rbd:vmlive/testguest-vda:auth_supported=none:mon_host=test1\:6789,if=none,id=drive-virtio-disk0,format=raw: error connecting
2016-01-06T15:16:54.640574Z qemu-kvm: -drive file=rbd:vmlive/testguest-vda:auth_supported=none:mon_host=test1\:6789,if=none,id=drive-virtio-disk0,format=raw: could not open disk image rbd:vmlive/testguest-vda:auth_supported=none:mon_host=test1\:6789: Could not open 'rbd:vmlive/testguest-vda:auth_supported=none:mon_host=test1\:6789': Operation not supported

Domain installation does not appear to have been successful.

If you've read this far… thanks, congratulations, and all help gratefully received!

Update 1

Update: This appears to be a problem with virt-install partially (but incompletely) parsing pools. If I use --print-xml the partial XML generated for the disk is:

<disk type="network" device="disk">
  <driver name="qemu"/>
  <source protocol="rbd" name="vmlive/testguest-vda">
    <host name="test1" port="6789"/>
  </source>
  <target dev="vda" bus="virtio"/>
</disk>

… which includes much of the pool definition, but not the auth stanza.

I'll keep investigating.