Shell – See extended attributes with coreutils ls on Mac

coreutilsosxshellxattr

I have coreutils installed via MacPorts on my Mac running OS X 10.8.4. I have ls set to use the coreutils version of ls [(GNU coreutils) 8.21] when available:

if [ -e /opt/local/libexec/gnubin ]; then
    alias ls='/opt/local/libexec/gnubin/ls --color=auto'
else
    alias ls='/bin/ls -G'
fi

When I run ls -l in a directory with files known to have extended attributes (xattrs), I expect to see an @ sign after the permissions in those listings. However, I see no @ sign. If I run /bin/ls -l, I get the @ sign.

File listing from /bin/ls -l:

-rw-r--r--@  1 zev.eisenberg  staff  132887 Jul 19 16:24 flowchart.graffle

File listing from ls -l (using coreutils):

-rw-r--r--  1 zev.eisenberg staff 132887 Jul 19 16:24 flowchart.graffle

How can I get the coreutils version of ls to show me the @ sign when xattrs are present?

Best Answer

You can add extended attributes to coreutils ls. This is based on coreutils-8.22:

***************
*** 59,62 ****
--- 59,64 ----
  #include <wchar.h>

+ #include <sys/xattr.h>
+
  #if HAVE_LANGINFO_CODESET
  # include <langinfo.h>
***************
*** 3056,3059 ****
--- 3058,3062 ----
                              : ACL_T_YES));
            any_has_acl |= f->acl_type != ACL_T_NONE;
+           any_has_acl |= listxattr(f->name, NULL, 0, XATTR_NOFOLLOW);

            if (err)
***************
*** 3811,3814 ****
--- 3814,3819 ----
    if (! any_has_acl)
      modebuf[10] = '\0';
+   else if (listxattr(f->name, NULL, 0, XATTR_NOFOLLOW) > 0)
+     modebuf[10] = '@';
    else if (f->acl_type == ACL_T_SELINUX_ONLY)
      modebuf[10] = '.';

Basically I looked in the OS X ls source to find the logic for printing the @ (the listxattr call) and hooked that into where coreutils ls puts a symbol after the permissions. The three changes are:

Including xattr.h
Set any_has_acl if any of the listings have extended attributes - this is required so that listings that don't have extended attributes have a space inserted after the permissions to line things up
Do the actual check by calling listxattr and conditionally setting the @ symbol - might be worth noting that they way this is written will show only the @ if there are both extended attributes and ACL

The XATTR_NOFOLLOW argument tells listxattr not to follow symlinks. That argument is used in OS X ls.

Related Solutions

Extended Attributes – Storage and Preservation

The answer to you question is filesystem specific. For ext3, for example, have a look at fs/ext3/xattr.c, it contains the following description:

  16 /*
  17  * Extended attributes are stored directly in inodes (on file systems with
  18  * inodes bigger than 128 bytes) and on additional disk blocks. The i_file_acl
  19 
 * field contains the block number if an inode uses an additional block. All
  20  * attributes must fit in the inode and one additional block. Blocks that
  21  * contain the identical set of attributes may be shared among several inodes.
  22  * Identical blocks are detected by keeping a cache of blocks that have
  23  * recently been accessed.
  24  *
  25  * The attributes in inodes and on blocks have a different header; the entries
  26  * are stored in the same format:
  27  *
  28  *   +------------------+
  29  *   | header           |
  30  *   | entry 1          | |
  31  *   | entry 2          | | growing downwards
  32  *   | entry 3          | v
  33  *   | four null bytes  |
  34  *   | . . .            |
  35  *   | value 1          | ^
  36  *   | value 3          | | growing upwards
  37  *   | value 2          | |
  38  *   +------------------+
  39  *
  40  * The header is followed by multiple entry descriptors. In disk blocks, the
  41  * entry descriptors are kept sorted. In inodes, they are unsorted. The
  42  * attribute values are aligned to the end of the block in no specific order.
  43  *
  44  * Locking strategy
  45  * ----------------
  46  * EXT3_I(inode)->i_file_acl is protected by EXT3_I(inode)->xattr_sem.
  47  * EA blocks are only changed if they are exclusive to an inode, so
  48  * holding xattr_sem also means that nothing but the EA block's reference
  49  * count can change. Multiple writers to the same block are synchronized
  50  * by the buffer lock.
  51  */

Regarding the "how are attributes connected" question, the link is in the other way round, the inode has a link to the extended attributes, see EXT3_XATTR_NEXT and ext3_xattr_list_entries in xattr.h and xattr.c respectively.

To recap, the attributes are linked to the inode and are fs dependent, so yes, you will lose the attributes when burning a CD rom or emailing a file.

Preserving extended attributes with cp/rsync

Update

After messing around with this some more and looking at the code for chattr and other e2fsprogs, it is clear that the attributes set by chattr and those set by libattr (eg with the command setfattr) are very different. chattr sets ext filesystem flags which simply do not map to an named attribute or namespace. None of them show up with any call to libattr's listxattr. They probably should map to named attributes in the system namespace as assumed below, but as of yet this is completely unimplemented. Also the system.posix_acl_access attribute I mistook for the mapping to one of these attributes below, is nothing to do with the ext filesystem flags and is rather to do with access control lists. The associated strace messages appear for any file and disappear when only cp --preserve=xattr is used.

It seems that the attributes set by chattr are specific to ext filesystems and that the only way to affect them is through e2fsprogs tools. In fact the man page does not actually use the term 'extended attributes' for them, but rather 'file attributes'. 'Real' extended attributes are name/value pairs which can be altered by libattr and are implemented on multiple filesystems. These are what cp and rsync look for and transfer over to copied files when the right options are given. It does however seem that system namespace exists to map the chattr attributes to names and ultimately to equivalent attributes on other filesystems, but for now this doesn't work.

I have left the original answer intact as there is some good information there, although it does go quite far wrong at points.

Update 2

I should have came back to this again before now, but as per this answer, chattr works on more than just ext filesystems. According to Wikipedia, it is equivalent to the chflags command on BSD based systems.

I wrote a script to test the setting and reading of these attributes on a few filesystems and got the following results:

ext4:
suS-iadAcj-t-e-- mnt/test_file
suSDiadAcj-tTe-- mnt/test_dir

reiserfs:
lsattr: Inappropriate ioctl for device While reading flags on mnt/test_file
lsattr: Inappropriate ioctl for device While reading flags on mnt/test_dir

xfs:
--S-iadA-------- mnt/test_file
--S-iadA-------- mnt/test_dir

btrfs:
--S-iadAc------C mnt/test_file
--SDiadAc------C mnt/test_dir

Note that all attempts to read/set reiserfs file flags gave the above error, despite it being listed on Wikipedia as having some functionality. I did not test reiser4. Also while the c flag can set on ext4 it is not honoured. There may also be tuning/mount options that affect these flags, but I couldn't find any.

It does however seem that currently chattr is the only utility on Linux capable of modifying these attributes and so no copy utility is capable of preserving them.

Original Answer

The reason for rsync seems to be that is doesn't even try. From the -X section of the rsync documentation:

For systems that support extended-attribute namespaces, a copy being done by a
super-user copies all namespaces except system.*.  A normal user only  copies
the user.* namespace.

It is difficult to map the attribute letters used by chattr and lsattr to the underlying named attributes used in the filesystem (for one there is no list on the internet). From my tests though, the A attribute maps to the system.posix_acl_access attribute and since this is the system namespace, rsync won't even try to copy it. The other two namespaces not mentioned in the man snippet are trusted and security, root privileges are required to set these (and rsync won't try without).

Most likely the attributes you have tried to set fall in the system namespace which rsync ignores (and probably wisely). Either that or you need to be root to get the ones that aren't.

~~As for cp, there appears to be bugs at play.~~ Running strace on cp -a, I get the following two interesting lines:

fgetxattr(3, "system.posix_acl_access", 0x7fff5181c0e0, 132) = -1 ENODATA (No data available)

and

fsetxattr(4, "system.posix_acl_access", "\x02\x00\x00\x00\x01\x00\x06\x00\xff\xff\xff\xff\x04\x00\x04\x00\xff\xff\xff\xff \x00\x04\x00\xff\xff\xff\xff", 28, 0) = 0

Firstly the fgetxattr call doesn't return any data (probably because there isn't any - the existence of the attribute is enough), yet somehow cp finds 28 bytes of (junk?) data to set as the attribute value in the destination file. This seems like a bug in cp, but rather what is causing the issues seems to be a bug in libattr as the fsetattr call returns 0 for success without actually setting the attribute.

I get this behaviour on ext4 regardless of whether I mount with user_xattr. I can't find any documentation on this other than to say that 'some systems' need this mount option for extended attributes to work. Seemingly mine (Debian Jessie) doesn't. Even there is a mounting issue I have missed, it is wrong for fsetattr and thus cp to fail silently.

Actually user_xattr is needed on ext2, ext3, reiserfs and possibly some others. It is not necessary for ext4

Note also that the attr tools setfattr, getfattr and attr (the latter is documented to be only for XFS only, but seems to work just as well as the others for ext4) have problems working in anything but the user namespace. I get Operation not supported if I try to use setfattr to put an attribute in the the system namespace (or no namespace as per this bug). setfattr appears to succeed in the trusted and security namespaces, but then getfattr fails to read anything back and also fails to read anything from the system namespace set by chattr. The reason that chattr succeeds is that it uses an ioctl call and not libattr.

What does work perfectly though, is setting extended attributes in the user namespace with setfattr and using rsync or cp to copy with them intact (there are even no issues with cp if you don't specify a value when creating the attribute). I think the bottom line is that using system namespace values is currently ~~buggy and/or~~ unsupported, at least in Debian and probably other distros too. Likely the rsync developers know this, which is why they ignore them.