btrfs will compress every file changed since it was mounted if you use:
mount -o compress-force=lzo /dev/btrfsdev /mnt/btrfsmnt
If you want to see to it that ALL files get compressed this way, I've got a little script I wrote to do it...
du -ht +$((1024*1024)) "$HOME" |\
sed -rn 's/^[^/]*(.*)/btrfs fi defrag -fvclzo "\1"/p' |\
sudo sh -n
The above only works on my $HOME directory - but you can use it on anything or everything as you like. It's also got the -n operand fed to sh as is so you can see for yourself what sh is currently not doing before removing it to tell it to do it.
Anyway, first it queries du for files in human-readable format (probably redundant here since we strip that in the next step anyway) that are larger than 1MB or $((1024*1024)).
It |pipes its info to sed which strips off everything before the leading /, "quotes" the filename, builds the btrfs filesystem defragment -verbose -flush-to-disk -compress-lzo \filename command and hands it over a |pipe to sudo sh to execute.
Again, it won't do anything so long as sh --no-execute is in effect, though.
I think some very recent btrfs userspace tool builds do defragment entire directories recursively, but if so it's a pretty new thing, so I've always had to do stuff like this.
In Debian/Ubuntu:
apt install btrfs-compsize
compsize /mnt/btrfs-partition
In Fedora:
dnf install compsize
compsize /mnt/btrfs-partition
output is like this:
Processed 123574 files, 1399139 regular extents (1399139 refs), 69614 inline.
Type Perc Disk Usage Uncompressed Referenced
TOTAL 73% 211G 289G 289G
none 100% 174G 174G 174G
lzo 32% 37G 115G 115G
Best Answer
It seems that only approach left is stacking a compressing filesystem on top of encrypting filesystem. Here are few options, but I do not have hands on experience with them:
I doubt that you can select compression or encryption per file. Btrfs is supposed to offer that but reality is probably far from it. Just like it was supposed to have RAID5 years ago.