Bash – What’s the difference between “>&1” and “>/proc/self/fd/1” redirection

bashio-redirectionsocket

I'm working on some script that being run by rc.local at startup, and I noticed that output redirection works quite strange.

If I write something like echo "foo" >&1, it ends up in syslog, and all is okay.

But when I write echo "foo" >>/dev/stdout, or echo "foo" >>/proc/self/fd/1, I got an error saying that there is no such device or address.

Further investigation revealed that /proc/self/fd/1 was in fact a socket.
Including ls -l /proc/self/fd in rc.local prints the following:

lr-x------. 1 root root 64 Jul 14 05:28 0 -> /dev/null
lrwx------. 1 root root 64 Jul 14 05:28 1 -> socket:[18485]
lrwx------. 1 root root 64 Jul 14 05:28 2 -> socket:[18485]
lr-x------. 1 root root 64 Jul 14 05:28 255 -> /etc/rc.d/rc.local

The described behaviour can be easily reproduced with nc. First, find out which file descriptor binds to socket. You can do it by issuing nc -l -p 25566 -c "ls -l /proc/self/fd" in first terminal and then telnet localhost 25566 in other terminal. In my case it was 5-th descriptor.

Okay. Then, to reproduce the issue, in first terminal:

nc -l -p 25566 -c "echo 'hello' >&5"

in second terminal:

telnet localhost 25566

You shall see "hello" in second terminal output between telnet messages about connection being established and closed.

Now the case with file descriptor pseudo-file from /proc:

first terminal:

nc -l -p 25566 -c "echo 'hello' >/proc/self/fd/5"

second terminal:

telnet localhost 25566

Now second terminal just contains telnet messages about connection established and immediately closed, and first terminal shows an error: sh: /proc/self/fd/5: No such device or address.

Edit: OS is Fedora 23 i686 server

So, the question. What's the difference between >&1 and >/proc/self/fd/1 ? And is there some universal and reliable way to redirect output to some file, which corresponds exactly to current standard output?

Thank you.

Edit 2:

For clarity, exact input/output for Fedora 23 i686 for above case:

Terminal 1                                       | Terminal 2
$ nc -l -p 25566 -c 'ls -go /proc/self/fd'       | 
                                                 | $ telnet localhost 25566
                                                 | Trying ::1...
                                                 | Connected to localhost.
                                                 | Escape character is '^]'.
                                                 | total 0
                                                 | lr-x------ 1 64 Jul 14 08:56 0 -> pipe:[19687]
                                                 | l-wx------ 1 64 Jul 14 08:56 1 -> pipe:[19688]
                                                 | lrwx------ 1 64 Jul 14 08:56 2 -> /dev/tty2
                                                 | lr-x------ 1 64 Jul 14 08:56 3 -> pipe:[19687]
                                                 | lr-x------ 1 64 Jul 14 08:56 4 -> /proc/1285/fd
                                                 | lrwx------ 1 64 Jul 14 08:56 5 -> socket:[19686]
                                                 | l-wx------ 1 64 Jul 14 08:56 7 -> /pipe:[19688]
                                                 | Connection closed by foreign host.
$ nc -l -p 25566 -c "echo 'hi' >&5"              | 
                                                 | $ telnet localhost 25566
                                                 | Trying ::1...
                                                 | Connected to localhost.
                                                 | Escape character is '^]'.
                                                 | hi
                                                 | Connection closed by foreign host.
$ nc -l -p 25566 -c "echo 'hi' >/proc/self/fd/5" | 
                                                 | $ telnet localhost 25566
                                                 | Trying ::1...
                                                 | Connected to localhost.
sh: /proc/self/fd/5: No such device or address   | Escape character is '^]'.
                                                 | Connection closed by foreign host.
$                                                | $

Best Answer

Use >&N. It's portable and as you saw, actually works with sockets.

The only reason you would ever use /proc/self/fd is you are running a program that expects a file name and can't be told to use an already open file descriptor. E.g. the <(cmd...) redirection uses that, since almost all command line utilities can open a file pointed to by name, but not all support preopened file descriptors directly.

Your shell can use pre-existing file descriptors, though, so no need to go through /proc.

Also, /proc/NNN/fd/ is specific to Linux, and requires that you have a mounted /proc. On my Linux boxes, /dev/stdout, /dev/fd/* and others are symlinks to /proc/self/fd/* etc., so they require /proc too. On other Unixes, they might be different. According to the answers to an older question /dev/stdout are specifically listed as outside POSIX.

As for why the redirection doesn't work as you tried: Trying out with strace, the difference between the two is that for a >&N redirection, bash calls dup() on the file descriptor, and for >/proc/self/fd/N it just tries to open it as an ordinary file with open(). Apparently proc doesn't support opening new copies of sockets like that, so the call fails. Stream sockets are pretty much point to point links, so prohibiting opening of a new copy doesn't seem too unnatural. But why it works for pipes or with a dup, I couldn't tell.

$ ls -l /proc/self/fd/3
lrwx------ 1 itvirta itvirta 64 Jul 14 18:24 /proc/self/fd/3 -> socket:[168157]
$ strace bash -c "echo foo >>/proc/self/fd/3" 2>&1 | grep open.*proc/self
open("/proc/self/fd/3", O_WRONLY|O_CREAT|O_APPEND, 0666) = -1 ENXIO (No such device or address)

Also this answer has some information about the portability of /proc/NNN/fd

Related Solutions

IO Redirection – Difference Between STDIN and Arguments Passed to Command

Your question is closely related to how the shell you are using parses user input on the command line.

If the first word on the command line is a program, located in a special folder (mostly defined by PATH) and no more special characters are given (depends of the shell you are using), all subsequent words separated by spaces or tabs are passed to the program in a special form i.e. an array. With each word as one element in the array.

How the program, you are going to invoke interprets the arguments (located in the array) depends on how it is programmed. There are some quasi standards of how the syntax of the arguments should look like but in general the programmer is entire free. So the first argument can be interpreted as a name of a file or whatever the programmer thoughts of at the time he wrote the program.

In the case you add the special character < or > to your command line, the shell dosn't append < and > nor subsequent words to the array that will be passed to the program. With < or > given the shell starts to make fancy things, supported by the underlying kernel (keyword piping). To grasp what's going on you must understand what STDIN and STDOUT (since it's not immediately related i omit STDERR) are.

Everything visible you see on your terminal (in most cases a part of your display) is either written by the shell or any other program you have invoked previously to a special file (in unix everything is a file). This file has a special id and is called STDOUT. If a program wants to read data from the keyboard it dosn't poll the keyboard directly (at least in most cases) but reads from a special file called STDIN. Internally this file is connected to your standard input device, your keyboard in most cases.

If the shell reads < or > in a parsed command line it manipulates STDIN or STDOUT in a particular kind for the time the corresponding program is running. STDIN and STDOUT dosn't point to the terminal or the standard input device any longer but rather to the subsequent filename on the command line.

In the case of the two lines

cat file_name
cat < file_name

the observed behavior is identical because the corresponding developer makes cat to either read data from STDIN or read the data from the file, whose name is given as the first command line argument (which is the first element in the array the shell passes to cat). Subsequently cat writes the whole content of file_name or STDIN to the terminal since we don't instruct the shell to manipulate STDOUT. Remember that in the second line your shell manipulates STDIN in this way, that it doesn't point to your standard input device anylonger but points to a file called file_name in your current working directory.

In the other case of the line

man < file_name

man is not meant to read anything from STDIN if it's called with no argument i.e. an empty array. So the line

man < file_name

equals

man

For example man will read something from STDIN, too if you pass -l - to man. With this option given on the command line you can display the content of anything man reads from STDIN on your terminal. So

man -l - < file_name

would work also (but be careful man is not just a pager but also parses the input of the file and so the file content and the displayed content could differ).

So how STDIN, STDOUT and the command line arguments are interpreted is all up to the corresponding developer.

I hope my answer could clear things up.

Shell – What’s the difference between sh script and sh < script

They are not. There are some things that will not work. The one thing that comes to mind is:

sh myscript -flag1 -value=3

is not possible with sh < myscript.

With sh < myscript, $0 is set to sh, rather than myscript, so

echo this script is $0

won't work

Also, if your script reads input, it will not work - you cannot

sh < myscript < myfiletoread

Best Answer

Related Solutions

IO Redirection – Difference Between STDIN and Arguments Passed to Command

Shell – What’s the difference between sh script and sh < script

Related Question