ZFS – zpool list vs zfs list: Why Free Space is 10x Different?

zfs

If I use zpool to list the available space, it tells me I have over 270 GB free, yet the actual free space available (and shown by df and zfs list) is only a mere 40 GB, almost ten times less:

$ zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ssdtank  7.25T  6.98T   273G        -         -    21%    96%  1.00x    ONLINE  -

$ zpool iostat -v
                                      capacity     operations     bandwidth 
pool                                alloc   free   read  write   read  write
----------------------------------  -----  -----  -----  -----  -----  -----
ssdtank                             6.98T   273G      1     15   365K   861K
  ata-Samsung_SSD_860_QVO_4TB_S123  3.49T   136G      0      7   182K   428K
  ata-Samsung_SSD_860_QVO_4TB_S456  3.49T   137G      0      8   183K   434K

$ zfs list
NAME       USED  AVAIL     REFER  MOUNTPOINT
ssdtank   6.98T  40.6G       32K  /srv/tank

What does this discrepancy mean? Why do the two utilities show differing amounts of free space? More importantly, how can I access the "extra" 200 GB if it's really there?

The pool is made of two identical disks, no RAID or other setups, just added as plain vdevs to the pool and the filesystem created on top. (There are multiple filesystems inside the root one shown, but I don't think they are relevant because they all share the same root and have the same 40.6G free space).

As requested, here is the output of zfs get all: (I also updated the figures above so they all make sense as the amount of free disk space has changed today. The old figures were 257GB/27GB and today they are 273GB/40GB, which means the amount of disk space freed up since I originally posted the question has increased both figures by the same amount – i.e. zpool seems to be reporting approx 270 GB more than everything else, but it's consistently 270 GB more than whatever the actual free space happens to be at the time).

NAME     PROPERTY              VALUE                  SOURCE
ssdtank  aclinherit            restricted             default
ssdtank  acltype               off                    default
ssdtank  atime                 off                    received
ssdtank  available             40.6G                  -
ssdtank  canmount              on                     default
ssdtank  casesensitivity       sensitive              -
ssdtank  checksum              on                     default
ssdtank  compression           off                    default
ssdtank  compressratio         1.00x                  -
ssdtank  context               none                   default
ssdtank  copies                1                      default
ssdtank  createtxg             1                      -
ssdtank  creation              Sat Oct 26 21:53 2019  -
ssdtank  dedup                 off                    default
ssdtank  defcontext            none                   default
ssdtank  devices               on                     default
ssdtank  dnodesize             legacy                 default
ssdtank  encryption            off                    default
ssdtank  exec                  on                     default
ssdtank  filesystem_count      none                   default
ssdtank  filesystem_limit      none                   default
ssdtank  fscontext             none                   default
ssdtank  guid                  12757787786185470931   -
ssdtank  keyformat             none                   default
ssdtank  keylocation           none                   default
ssdtank  logbias               latency                default
ssdtank  logicalreferenced     16K                    -
ssdtank  logicalused           6.98T                  -
ssdtank  mlslabel              none                   default
ssdtank  mounted               yes                    -
ssdtank  mountpoint            /srv/tank              local
ssdtank  nbmand                off                    default
ssdtank  normalization         none                   -
ssdtank  objsetid              54                     -
ssdtank  overlay               off                    default
ssdtank  pbkdf2iters           0                      default
ssdtank  primarycache          all                    default
ssdtank  quota                 none                   default
ssdtank  readonly              off                    default
ssdtank  recordsize            128K                   default
ssdtank  redundant_metadata    all                    default
ssdtank  refcompressratio      1.00x                  -
ssdtank  referenced            32K                    -
ssdtank  refquota              none                   default
ssdtank  refreservation        none                   default
ssdtank  relatime              off                    default
ssdtank  reservation           none                   default
ssdtank  rootcontext           none                   default
ssdtank  secondarycache        all                    default
ssdtank  setuid                on                     default
ssdtank  sharenfs              rw=@192.168.0.0/24     received
ssdtank  sharesmb              off                    default
ssdtank  snapdev               hidden                 default
ssdtank  snapdir               hidden                 default
ssdtank  snapshot_count        none                   default
ssdtank  snapshot_limit        none                   default
ssdtank  special_small_blocks  0                      default
ssdtank  sync                  standard               default
ssdtank  type                  filesystem             -
ssdtank  used                  6.98T                  -
ssdtank  usedbychildren        6.98T                  -
ssdtank  usedbydataset         32K                    -
ssdtank  usedbyrefreservation  0B                     -
ssdtank  usedbysnapshots       0B                     -
ssdtank  utf8only              off                    -
ssdtank  version               5                      -
ssdtank  volmode               default                default
ssdtank  vscan                 off                    default
ssdtank  written               0                      -
ssdtank  xattr                 on                     default
ssdtank  zoned                 off                    default

Best Answer

Internally ZFS reserves a small amount of space (slop space) to ensure some critical ZFS operations can complete even in situations with very low free space.

The amount is 3.2% of total pool capacity. zfs-0.6.5

3.2% of 7.25T = 235GB

You really only have 40.6GB free in the filesystem.

zpool reports about the raw disk capacities and the free space will be 40 + 235 = 275G

Related Solutions

Zpool import – cannot import: one or more devices is currently unavailable

Apparently there is incompatibility between Solaris 11 and Solaris 10 - ZFS Encryption and ZFS Deduplication. Although they are supported from ZFS version 31 and 21 respectively on Solaris 11 they aren't supported on Solaris 10 at all even though it supports up to version 32 :(

Hint: zpool upgrade -v for a list of supported features.

Linux – Does ZFS for Linux over stress VirtualBox

These look like generic hdd timeout errors in the guest system. They might be caused by ZFS, but they might just as well be caused by other high i/o operations. As a guest system, Linux is quite sensitive in this regard, as it has a low default timeout (usually 30 seconds). This may not be enough in a vm, especially if the disk image is a regular file and the host system is under load; some writes could take longer than expected if the host's cache is full.

Or, to quote the VirtualBox manual:

However, some guests (e.g. some Linux versions) have severe problems if a write to an image file takes longer than about 15 seconds. Some file systems however require more than a minute to complete a single write, if the host cache contains a large amount of data that needs to be written.

Note that this is not limited to VirtualBox. Other virtualization solutions may show the same behavior when running a Linux guest.

As for the timeout itself: The Linux hdd timeout (leading to ata exceptions and possibly corruption under high load) can be increased in the guest system.

For example, on Debian 7, all you need to do is add a few lines to your /etc/rc.local:

$ cat /etc/rc.local 
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

TIMEOUT=86400
for f in /sys/block/sd?/device/timeout; do
    echo $TIMEOUT >"$f"
done

exit 0

Then grep for ata exceptions to see if they're gone:

# grep -Rn --col 'ata.*exception' /var/log/

However, it would be preferable to increase the vm's disk performance rather than having to change the timeout of the guest system. In the case of VirtualBox, the "Host I/O Cache" of the vm's virtual storage controller can be disabled. If enabled, the host cache could be the bottleneck and slow disk operations down if there's a lot of disk i/o on the host. On the other hand, disabling it might increase the load on the vm itself so timeouts might still occur if the guest is overloaded, so enabling the host cache might even be better in some cases, depending on your workload.

If this does not help, the VirtualBox manual also recommends experimenting with the flush interval:

For IDE disks use the following command:

VBoxManage setextradata "VM name"
  "VBoxInternal/Devices/piix3ide/0/LUN#[x]/Config/FlushInterval" [b]

For SATA disks use the following command:

VBoxManage setextradata "VM name"
  "VBoxInternal/Devices/ahci/0/LUN#[x]/Config/FlushInterval" [b]

Values between 1000000 and 10000000 (1 to 10 megabytes) are a good starting point. Decreasing the interval both decreases the probability of the problem and the write performance of the guest.

In some tests, VirtualBox guest systems have experienced such hdd timeouts (crashing the vm and/or causing corruption) no matter if host i/o caching was enabled or not. The host filesystem was not slow, except for half a minute whenever a scheduled cron job would run, causing those timeouts in the vm. It was only after setting the hdd timeout as described above that the issue went away and no more timeouts happened.

Best Answer

Related Solutions

Zpool import – cannot import: one or more devices is currently unavailable

Linux – Does ZFS for Linux over stress VirtualBox

Related Question