Error in R: Unable to open connection to X11

Since $DISPLAY is correctly set and the ~/.Xauthority file is not created, this can mean that, though X11 forwarding is taken into account, xauth is not run. One reason could be that it is not in the path (I had this problem under Mac OS X, but this would be strange under Linux). You may want to do the work yourself by creating a ~/.ssh/rc file. For instance, I have the following:

if [ -n "$DISPLAY" ]; then
  echo "DISPLAY: $DISPLAY" >&2
  if read proto cookie; then
    if [ `echo $DISPLAY | cut -c1-10` = 'localhost:' ]; then
      echo add unix:`echo $DISPLAY | cut -c11-` $proto $cookie
    else
      echo add $DISPLAY $proto $cookie
    fi | $HOME/.ssh/xauth.wrapper -q -
  fi
fi

where ~/.ssh/xauth.wrapper is a wrapper to xauth that implements the locking of the ~/.Xauthority file. But you can use just xauth or the full pathname to xauth just in case... This is much like what is described in the sshd(8) man page (see Section "SSHRC").

Be careful not to do any mistake.

Xauthority Mini How To

On GNU/Linux systems running an X11 display server, the file ~/.Xauthority stores authentication cookies or cryptographic keys used to authorize connection to the display. In most cases, the authentication mechanism is a symmetric cookie which is referred to as a Magic Cookie. The same cookie is used by the server as well as the client.

Each X11 authentication cookie is under the control of the individual system authenticated user. Since the authetication cookie is stored as a plain text security token, the permissions on the ~/.Xauthority file should be rw for the owner only, 600 in octal format. However, the permissions on the authorization file are not enforced.

A user can list, export, create, or delete authentication cookies using the xauth program. The following command will create an authoratization cookie for DISPLAY 32.

xauth add localhost:32 - `mcookie`

Manual creation and manipulation of cookies is usually not needed when using X11 forwarding with ssh, because ssh starts an X11 proxy on the remote machine and automatically generates authorization cookies on the local display. However, for certain configurations the authorization cookie may need to be manually created and copied to the local machine.

This can be done in an ssh session and then use scp to copy the cookie.

ssh into remote machine:

ssh -XY user@remote

Check if an authorization cookie is present for the current X11 display

echo $DISPLAY
xauth list

If there's no environment variable named $DISPLAY then the X11 proxy did not start properly. It's important to note that DISPLAY 0 is typically locally logged in users and is only running if an xserver has been locally started via xinit. There is no requirement for a locally started X11 server in order for X11 forwarding to function through ssh.

If there's a $DISPLAY environment variable set but no corresponding authorization cookie for that display number, you can create one:

xauth add $DISPLAY - `mcookie`

And verify that there is now a cookie:

xauth list

You can copy that cookie and merge it into the local machine:

user@remote> xauth nextract ~/xcookie $DISPLAY
user@remote> exit
user@local> scp user@remote:~/xcookie ~/xcookie
user@local> xauth nmerge ~/xcookie

And then verify that the cookie has been installed:

user@local> xauth list

Try out your X11 forwarding ssh connection.

Notes on ~/.Xauthority

~/.Xauthority is a binary file which contains all the authorization information for each display the user may access. Each record is delimited by the two bytes 0x0100. Each field is preceeded by a hexidemical count of the field's number of bytes. All text is encoded in hexidecimal ASCII. The following table is the basic structure of the most common configuration of a MIT MAGIC COOKIE authorization:

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 0100              0004        61616161           0002           3435                    0012          4d49542d4d414749432d434f4f4b49452d31   0010          c0bdd1c539be89a2090f1bbb6b414c2c 
----------------- ----------- ------------------ ------------  ----------------------  -------------  -------------------------------------- ------------ ---------------------------------------
 start-of-record   0xNumBytes  0xASCII Hostname   0xNumBytes     0xASCII Display Num     0xNumBytes    0xASCII Auth Type                      0xNumBytes    0xkey
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The top line is retrievable from the ~/.Xauthority file via the xauth nlist command. Of course, your authorization file will have different information from my example.

If the Security Extensions are in use with the X11 server, there are several configuration options for each authorization line including time limited authorization per cookie.