How to expand btrfs metadata

TL;DR The metadata (if the btrfs is not suffering general low space condition) will automatically increase. In cases that no unallocated free space exists, the automatic increase is hembered. If, however, the data part of btrfs has been allocated more space than it needs, then it is possible to redistribute this. This is called balance-ing in btrfs.

Assuming that there is enough unallocated memory on the backing block device(s) of the btrfs, then the Metadata part of the filesystem allocates - just as assumed by the OP - automatically memory to increase/expand the metadata.

Therefore, the answer is: Yes (provided there is not low memory/free space condition in the btrfs), then the metadata will get automatically increased, as such:

(1) We have a look at some initial allocation setup of the btrfs (on a 40GB device)

$> btrfs filesystem df /
Data, single: total=25.00GiB, used=24.49GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.55GiB, used=1.33GiB
GlobalReserve, single: total=85.41MiB, used=0.00B

(2) As can be seen, the allocated space in the filesystem to store Metadata is 1.55GiB, of which 1.33GiB, hence almost all is used (this might be a situation as occurring in the OP's case)

(3) We now provoke an increase of metadata to be added. To do so, we copy the /home folder using the --reflink=always option of the cp command.

$> cp -r --reflink=awlways /home /home.copy

(4) Since (as we assume there were lots of files in /home), which a lot of new data to the filesystem has been added, which because we used --reflink does use little to no additional space for the actual data, it uses the Copy-on-Write, mechanism. In short, mostly Metadata was added to the filesystem. We can have hence another look

$> btrfs filesystem df /
Data, single: total=25.00GiB, used=24.65GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=2.78GiB, used=2.45GiB
GlobalReserve, single: total=85.41MiB, used=0.00B

As can be seen, the space allocated for Metadata used in this btrfs has automatically increased expanded.

Since this is so automatically, it normally goes undetected by the user. However, there are some cases, mostly those where the whole filesystem is already pretty much filled up. In those cases, btrfs may begin to "stutter" and fail to automatically increase the allocated space for the Metadata. The reason would be, for example, that all the space has already been allocated to the parts (Data, System, Metadata, GlobalReserve). Confusingly, it could be yet the case that there is apparent space. An example would be this output:

$> btrfs filesystem df /
Data, single: total=38.12GiB, used=25.01GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.55GiB, used=1.45GiB
GlobalReserve, single: total=85.41MiB, used=0.00B

As it can be seen, the system all the 40GiB, yet the allocation is somewhat off balance, since while there is still space for the new files' data, the Metadata (as in the OP case) is low. The automatic allocation of memory for the devices backing the btrfs filesystem is not anymore possible (simply add up the totals of the allocation, 38.12G+1.55G+..~= 40GiB).

Since there is however excess free space that was allocated to the data part of the filesystem, it can now be useful, necessary to balance the btrfs. Balance would mean to redistribute the already allocated space.

In the case of the OP, it may be assumed that, for some reason, an imbalance between the different parts of the btrfs allocation has occurred.

Unfortunately, the simple command sudo btrfs balance -dusage=0, which in principle should search empty blocks (allocated for data) and put them to better user (that would be the almost depleted space for Metadata), may fail, because no completely empty data blocks can be found.

The btrfs developers recommend to hence successively increase the usage limit of "when data blocks should be rearranged to reclaim space"

Hence, if the result of

$> sudo btrfs balance -dusage=0
Done, had to relocate 0 out of 170 chunks 

is showing no relocation, one should do some

$> sudo btrfs balance -dusage=5
Done, had to relocate 0 out of 170 chunks  <--(again fail)
$> sudo btrfs balance -dusage=10
Done, had to relocate 0 out of 170 chunks  <--(again fail)
$> sudo btrfs balance -dusage=15
Done, had to relocate 2 out of 170 chunks  <--(success)

The other answer has hinted on the influence of the btrfs nodesize, which influences somewhat how quickly metadata will increase. The nodesize is (as mentioned in the other answer) only set once at mkfs.btrfs filesystem creation time. In theory, one could reduce the size of Metadata if it was possible to change to a lower value for the nodesize, if that was possible (it is not!). The nodesize, however, will not be able to help expand or increase the metadata space allocated in any way. Instead, it might have only helped to conserve space in the first place. A smaller nodesize, is not guaranteed however to reduce the metadata size. Indeed, some cases might show that larger nodesizes reduce the tree-traversal length of btrfs, as notes can contain more "links".