Am I correct in the understanding of how symlinks and `..` interact under POSIX

filenamesposixsymlink

I am working on a mathematical description of paths (such as filepaths, but also more abstract and general)

One of the most tricky things to define is the behavour of .. (φ in the linked post); particularly with how it interacts with symlinks.

I want to check that for it that I understand the POSIX rules correctly.

POSIX 4.14 says

When a process resolves a pathname of an existing directory entry, the entire pathname shall be resolved as described below. When a process resolves a pathname of a directory entry that is to be created immediately after the pathname is resolved, pathname resolution terminates when all components of the path prefix of the last component have been resolved. It is then the responsibility of the process to create the final component. …
Each filename in the pathname is located in the directory specified by its predecessor (for example, in the pathname fragment a/b, file b is located in directory a).
…
If a symbolic link is encountered during pathname resolution, …
the system shall prefix the remaining pathname, if any, with the contents of the symbolic link …
the resolved pathname shall be the resolution of the pathname just created. If the resulting pathname does not begin with a , the predecessor of the first filename of the pathname is taken to be the directory containing the symbolic link.
The special filename dot shall refer to the directory specified by its predecessor. The special filename dot-dot shall refer to the parent directory of its predecessor directory. As a special case, in the root directory, dot-dot may refer to the root directory itself. ..

So what I understand is POSIX says is that symlinks should expanded before resolving .. to go to the parent directory.

Is this correct?

So for normalising paths, the POSIX compliant way is the behaviour of
realpath -P
which is to require existence of all directories (but not the final file component), and to expand resolve symlinks from left to right (as soon as encountered), and then apply ..
Which is to say the file system has to be read at all steps of processing a path.

We can contrast this to the behaviour of node.js's Path.normalize (or even it's Path.posix.normalize) — which I think is fairly normal in many programming langauges (Python2 and 3 os.path are similar).
Which is equivalent to realpath -s -m, which is to say it completely ignores that some directories may be simlinks, or may not exist.
Which is nice since it does not have to touch the file system at all.

If we than take a path that has been normalised this way,
and give it to a function that does touch the file system (Eg fs.readFile)
Then it is equivalent to if we had normalized the path using realpath -L,
which is to "Resolve .. before symlinks"

Bash cd also acts as though it has processed it argument with realpath -L unless you give it the -P flag

Am I correctly understanding the POSIX spec, and its relation to realpath, and to many (most?) programming language path libraries?

Bonus question :-P, anyone got an example of a programming languages/libraries (other than using shell utilities like realpath) that has a POSIX compliant implementation of normalize?
(I believe Python 3 pathlib.Path.resolve is correct. Though it also converts relative paths to absolute paths.)

Best Answer

. and .. on Unixes are real directory entries that represent real files (files or directories) that belong to their parent directory.

.. isn't a path resolution keyword to strip the last segment of the path. In order to resolve a .., you first need to resolve the directory that contains it. Nodejs is doing it wrong.

$ mkdir -p /tmp/foo/bar/baz
$ ln -s /tmp/foo/bar/baz /tmp/symlink
$ realpath /tmp/symlink/..
  /tmp/foo/bar
$ node -e 'console.log(path.normalize("/tmp/symlink/.."))'
  /tmp

(These entries are always created when a directory is created, and they are made to be hardlinks to the current directory and to its parent. They are some of the rare examples of directory hardlinks, as systems won't normally allow regular users to make directory hardlinks.

That's why the hardlink count in ls -ld listings increases for a directory whenever you make a subdirectory, and why directories start at hardlink count 3 (their name, ., and their_name/..). The subdirectory's .. is a hardlink to the current directory and thereby increases its hardlink count. )

Related Solutions

Linux – How to efficiently dereference all symlinks in `find` output filenames

What you're asking for doesn't make much sense in the general case, so it's not surprising that find has no provision for it.

A symlink with a relative target is relative to the path of the symlink. So for instance, if by traversing a directory by following symlinks, find encounters a/b/c/d and a, a/b, a/b/c are all relative or absolute symlinks (or symlinks to paths with symlink components), what should it do?

If you're looking for a find predicate or a GNU -printf % directive that expands to a symlink-free path to the file relative to the current directory or any directory, I'm afraid there's none.

If you're on Linux, you can get the absolute path of those files with:

find -L foo -type f -exec readlink -f {} \;

As you found out, there exists at least one realpath command which accepts more than one path argument which in combination with the standard -exec cmd {} + syntax is going to be a lot more efficient since it's running as few realpath commands as necessary:

find -L foo -type f -exec realpath {} +

find -L foo -type f -print0 | xargs -r0 realpath

might be quicker as if more than one realpath command is needed, find can keep on looking for more files while the first realpath starts working which even on a single processor system might make it more efficient.

-print0 and xargs -r0 are not standard, come from GNU but are found in a number of other implementations like most modern BSDs.

Zsh has builtin support for it:

print -rl foo/***/*(-.:A)

If you don't care about the sorting order, you can disable sorting and make it a bit more efficient with:

print -rl foo/***/*(-.oN:A)

If you want to convert those to relative paths to the current directory, you could have a look at that SO question.

If you know that all those files have an absolute canonical path (whose none of the components are symlinks) inside the current directory, you can simplify it to (still with zsh):

files=(foo/***/*(-.:A))
print -rl -- ${files#$PWD/}

Though short and convenient, and works whatever character filenames contain, I doubt it would faster than find + realpath.

With the Debian realpath and GNU tools, you can do:

cd -P .
find -L foo -type f -exec realpath -z {} + | 
  gawk -v p="$PWD" -v l="${#PWD}" -v RS='\0' -vORS='\0' '
    substr($0, 1, l+1) == p "/" {$0 = substr($0, l+2)}; 1' |
  xargs -r0 whatever you want to do with them

As I realise now, there's now a realpath in recent versions of GNU coreutils, which has the exact feature you're looking for, so it's just a matter of

find -L foo -type f -print0 |
  xargs -r0 realpath -z --relative-base . |
  xargs -r0 whatever you want to do with them

(use --relative-to . instead of --relative-base . if you want relative paths even for files whose symlink free path doesn't reside below the current working directory).

Shell – `/proc/$PID/cwd` : is there a POSIX equivalent

I have a solution that uses lsof. It is not installed on BSD by default so if anyone want to use it on BSD, it is required to install it.

Make a shell script:

#!/bin/sh
lsof -p $1 | grep cwd | awk '{print $9}'

Copy it to a directory in your path. It prints the working directory of PID given in the first argument, I.E.

$ script 1987
/home/enedil

Best Answer

Related Solutions

Linux – How to efficiently dereference all symlinks in `find` *output* filenames

Shell – `/proc/$PID/cwd` : is there a POSIX equivalent

Related Question

Linux – How to efficiently dereference all symlinks in `find` output filenames