While trying to mount a NFS share (exported from an OpenIndiana server) on a client box, the OI server crashed. I got the black screen of death, what looked like a log dump, then the system restated. It never came back up and I get the following error msg after I halt the boot:
svc.startd[9] Could not log for svc:/network/dns/mulitcast:default: write(30) failed with No space left on device?
I don't have anything else on the boot drive other than the OS so… I'm not sure what could be filling up the drive? Maybe a log file of some kind? I can't seem to delete anything regardless. It gives me a no space error when I try and delete anything:
$ rm filename
cannot remove 'filename' : No space left on device
I can login into "Maintenance Mode" but not the standard user prompt.
The output of df is:
rpool/ROOT/openindiana-baseline 4133493 4133493 0 100% /
swap 83097900 11028 830386872 1% /etc/svc/volatile
/usr/lib/libc/libc_hwcap1.so.1 4133493 4133493 0 100% /lib/libc.so.1
The output of mount is:
/ on rpool/ROOT/openindiana-baseline read/write/setuid/devices/dev:2d9002 on Wed Dec 31 16:00:00 1969
/devices on /devices read/write/setuid/devices/dev:8b40000 on Fri Jul 8 14:56:54 2011
/dev on /dev read/write/setuid/devices/dev:8b80000 on Fri Jul 8 14:56:54 2011
/system/contract on ctfs read/write/setuid/devices/dev:8c40001 on Fri Jul 8 14:56:54 2011
/proc on proc read/write/setuid/devices/dev:8bc0000 on Fri Jul 8 14:56:54 2011
/etc/mnttab on mnttab read/write/setuid/devices/dev:8c80001 on Fri Jul 8 14:56:54 2011
/etc/svc/volatile on swap read/write/setuid/devices/xattr/dev:8cc0001 on Fri Ju8 14:56:54 2011
/system/object on objfs read/write/setuid/devices/dev:8d00001 on Fri Jul 8 14:6:54 2011
/etc/dfs/sharetab on sharefs read/write/setuid/devices/dev:8d40001 on Fri Jul 14:56:54 2011
/lib/libc.s0.1 on /usr/lib/libc/libc_hucap1.s0.1 read/write/setuid/devices/dev:d90002 on Fri Jul 8 14:57:06 2011
The output of 'zfs list -t all' is:
rpool 36.4G 0 47.5K /rpool
rpool/ROOT 4.23G 0 31K legacy
rpool/ROOT/openindiana 57.5M 0 3.99G /
rpool/ROOT/openindiana-baseline 61K 0 3.94G /
rpoo1/ROOT/openindiana-system-edge 4.17G 0 3.98G /
rpool/ROOT/openindiana-system-edge@install 19.9M - 3 38G -
rpoo1/ROOT/openindiana-system-edge@2011-07-06-20:45:08 73.1M - 3.57G -
rpoo1/ROOT/openindiana-system-edge@2011-07-06-20:48:53 75.9M - 3 82G -
rpoo1/ROOT/openindiana-system-edge@2011-07-07-02:14:04 61K - 3.94G -
rpoo1/ROOT/openindiana-system-edge@2011-07-07-02:15:14 61K - 3.94G -
rpoo1/ROOT/openindiana-system-edge@2011-07-07-02:28:14 61K - 3.94G -
rpool/ROOT/openindiana-system-stable 61K 0 3.94G /
rpoo1/ROOT/pre_first_update_07.06 108K 0 3 82G /
rpool/ROOT/pre_second_update_07.06 90K 0 3.57G /
rpool/dump 9.07G 0 9.07G -
rpool/export 3.85G 0 32K /export
rpool/export/home 3.85G 0 32K /export/home
rpool/export/home/admin 3.85G 0 3.85G /export/home/admin
rpool/swap 19.3G 19.1G 126M -
Best Answer
Ok, that's a weird one… not enough space to remove a file!
This turns out to be a relatively common issue with ZFS, though it could potentially arise on any filesystem that has snapshots.
The explanation is that the file you're trying to delete still exists on a snapshot. So when you delete it, the contents keep existing (in the snapshot only); and the system must additionally write the information that the snapshot has the file but the current state doesn't. There's no space left for that extra little bit of information.
A short-term fix is to find a file that's been created after the latest snapshot and delete it. Another possibility is to find a file that's been appended to after the latest snapshot and truncate it to the size it had at the time of the latest snapshot. If your disk got full because something's been spamming your logs, try trimming the largest log files.
A more generally applicable fix is to remove some snapshots. You can list snapshots with
zfs list -t snapshot. There doesn't seem to be an easy way to predict how much space will be regained if you destroy a particular snapshot, because the data it stores may turn out to be needed by other snapshots and so will live on if you destroy that snapshot. So back up your data to another disk if necessary, identify one or more snapshots that you no longer need, and runzfs destroy name/of/snap@shot.There is an extended discussion of this issue in this OpenSolaris forums thread.