How to find out more about socket files in /proc/fd

I believe that the idea of the socket being unavailable to a program is to allow any TCP data segments still in transit to arrive, and get discarded by the kernel. That is, it's possible for an application to call close(2) on a socket, but routing delays or mishaps to control packets or what have you can allow the other side of a TCP connection to send data for a while. The application has indicated it no longer wants to deal with TCP data segments, so the kernel should just discard them as they come in.

I hacked out a little program in C that you can compile and use to see how long the timeout is:

#include <stdio.h>        /* fprintf() */
#include <string.h>       /* strerror() */
#include <errno.h>        /* errno */
#include <stdlib.h>       /* strtol() */
#include <signal.h>       /* signal() */
#include <sys/time.h>     /* struct timeval */
#include <unistd.h>       /* read(), write(), close(), gettimeofday() */
#include <sys/types.h>    /* socket() */
#include <sys/socket.h>   /* socket-related stuff */
#include <netinet/in.h>
#include <arpa/inet.h>    /* inet_ntoa() */
float elapsed_time(struct timeval before, struct timeval after);
int
main(int ac, char **av)
{
        int opt;
        int listen_fd = -1;
        unsigned short port = 0;
        struct sockaddr_in  serv_addr;
        struct timeval before_bind;
        struct timeval after_bind;

        while (-1 != (opt = getopt(ac, av, "p:"))) {
                switch (opt) {
                case 'p':
                        port = (unsigned short)atoi(optarg);
                        break;
                }
        }

        if (0 == port) {
                fprintf(stderr, "Need a port to listen on\n");
                return 2;
        }

        if (0 > (listen_fd = socket(AF_INET, SOCK_STREAM, 0))) {
                fprintf(stderr, "Opening socket: %s\n", strerror(errno));
                return 1;
        }

        memset(&serv_addr, '\0', sizeof(serv_addr));
        serv_addr.sin_family      = AF_INET;
        serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
        serv_addr.sin_port        = htons(port);

        gettimeofday(&before_bind, NULL);
        while (0 > bind(listen_fd, (struct sockaddr *)&serv_addr, sizeof(serv_addr))) {
                fprintf(stderr, "binding socket to port %d: %s\n",
                        ntohs(serv_addr.sin_port),
                        strerror(errno));

                sleep(1);
        }
        gettimeofday(&after_bind, NULL);
        printf("bind took %.5f seconds\n", elapsed_time(before_bind, after_bind));

        printf("# Listening on port %d\n", ntohs(serv_addr.sin_port));
        if (0 > listen(listen_fd, 100)) {
                fprintf(stderr, "listen() on fd %d: %s\n",
                        listen_fd,
                        strerror(errno));
                return 1;
        }

        {
                struct sockaddr_in  cli_addr;
                struct timeval before;
                int newfd;
                socklen_t clilen;

                clilen = sizeof(cli_addr);

                if (0 > (newfd = accept(listen_fd, (struct sockaddr *)&cli_addr, &clilen))) {
                        fprintf(stderr, "accept() on fd %d: %s\n", listen_fd, strerror(errno));
                        exit(2);
                }
                gettimeofday(&before, NULL);
                printf("At %ld.%06ld\tconnected to: %s\n",
                        before.tv_sec, before.tv_usec,
                        inet_ntoa(cli_addr.sin_addr)
                );
                fflush(stdout);

                while (close(newfd) == EINTR) ;
        }

        if (0 > close(listen_fd))
                fprintf(stderr, "Closing socket: %s\n", strerror(errno));

        return 0;
}
float
elapsed_time(struct timeval before, struct timeval after)
{
        float r = 0.0;

        if (before.tv_usec > after.tv_usec) {
                after.tv_usec += 1000000;
                --after.tv_sec;
        }

        r = (float)(after.tv_sec - before.tv_sec)
                + (1.0E-6)*(float)(after.tv_usec - before.tv_usec);

        return r;
}

I tried this program on 3 different machines, and I get a variable time, between 55 and 59 seconds, when the kernel refuses to allow a non-root user to reopen a socket. I compiled the above code to an executable named "opener", and ran it like this:

./opener -p 7896; ./opener -p 7896

I opened another window and did this:

telnet otherhost 7896

That causes the first instance of "opener" to accept a connection, then close it. The second instance of "opener" tries to bind(2) to the TCP port 7896 every second. "opener" reports 55 to 59 seconds of delay.

Googling around, I find that people recommend doing this:

echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout

to reduce that interval. It didn't work for me. Of the 4 linux machines I had access to, two had 30 and two had 60. I also set that value as low as 10. No difference to the "opener" program.

Doing this:

echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle

did change things. The second "opener" only took about 3 seconds to get its new socket.

In most environments, you would only expect to find tcp, udp, raw and packet sockets. Happily, ss knows about all of these.

Assuming ss knows all the protocols you need it to, I might use the following command. This will exclude the "unix" sockets. It also excludes "netlink" sockets, which are only used to communicate with the local kernel.

sudo ss -l -p | grep -vE '^(u_|nl )'

Often you do not have a lot of listening sockets. So you can look through them all, and manually ignore any that listen on loopback IP addresses. Alternatively, you can ask ss to do all the filtering:

sudo ss -l -p -A 'all,!unix,!netlink' 'not src 127.0.0.1 not src [::1]'

In both cases, the output can also include "client" udp sockets. So it might also show DNS clients, HTTP/3 clients, ...

If you do not need to see information about the program which opened each socket, then you can remove the -p option, and you do not need to run ss (or netstat) with root privileges (sudo).

How comprehensive is the above command?

Despite being advertised as a replacement for netstat, ss lacks the support for showing "udplite" sockets.

Also, the answer depends on your version of ss (and I guess the kernel as well). When this answer was originally written, before 2017, ss did not support "sctp". netstat supported it since February 2014). sctp is expected specifically inside phone companies. Outside of phone companies, VOIP typically uses udp.

Unfortunately if you look for a comprehensive list in man netstat, it gets quite confusing. Options for sctp and udplite are shown in the first line, along with tcp, udp and raw. Further down there's what looks like a comprehensive list of protocol families: [-4|--inet] [-6|--inet6] [--unix|-x] [--inet|--ip|--tcpip] [--ax25] [--x25] [--rose] [--ash] [--bluetooth] [--ipx] [--netrom] [--ddp|--appletalk] [--econet|--ec].

Although netstat supports udplite and sctp, it does not support "dccp". Also netstat doesn't support packet sockets (like raw sockets but including link-level headers), as selected by ss -l -0. In conclusion, I hate everything, and I could probably stand to be less pedantic.

Also ss does not support bluetooth sockets. Bluetooth sockets are not a traditional concern. This could be relevant if you were doing a full audit. Bluetooth security is quite a specific question though; I am not answering it here.

Omitting localhost in netstat?

netstat does not have a specific way to omit sockets bound to localhost. You could use | grep -v on the end. Take care if you use the -p option to netstat / ss. You might accidentally exclude some of your processes, if there is a match in the process name. I would include the colon in your pattern, like grep -v localhost:. Except the default in ss is to show numeric addresses, so in that case you would use | grep -vE (127.0.0.1|\[::1\]):. I suppose you could try to check for processes which would be accidentally excluded, e.g. ps ax | grep -E (127.0.0.1|\[::1\]):.

Is there a simpler command?

It's unfortunate about the packet sockets. Otherwise, I might suggest a plain netstat -l command. netstat helpfully anticipates your request and splits the output into "Internet connections", "UNIX domain sockets", and "Bluetooth connections". You would just look at the first section. There is no section for netlink sockets.

Suppose you're only concerned with tcp, udp, raw, and packet sockets. For the first three types of socket you could use netstat -l -46.

Packet sockets are in common use. So you would also need to train yourself to run ss -l -0 (or ss -l --packet).

Unfortunately this leaves you with a big pitfall. The problem is it is now tempting to try and combine the two commands...

Traps to avoid with ss

ss -l -046 looks appealing as a single-command answer. However this is not true. ss -46 only shows IPv6 sockets. ss -64 only shows IPv4 sockets.

I suggest always sanity-checking your results. Learn what to expect; go through each protocol and see if there's anything missing that should be there. If you have no IPv4 addresses, or no IPv6 addresses, that's very suspicious. You can expect most servers to have an SSH service listening on both. Most non-servers should also show packet or raw sockets, due to using DHCP.

If you don't want to interpret the output of two different commands, one alternative might be to replace the netstat command with ss -l -A inet. This is slightly unfortunate because when you run netstat, the exact same options would exclude ipv6 sockets.

So for a single command, you could use ss -l -A inet,packet ....

IMO you might as well use ss -l | grep ... as I suggested in the first section. It is easy to remember this command, and it avoids any and all confusing behaviour in the selection options of ss.

Although if you write scripts that use this output to automate something, then you should probably prefer to filter on a positive list of socket types instead. Otherwise the script could break when ss starts supporting a new type of local-only socket.

Did I mention that ss -a -A raw -f link shows a combination of sockets from ss -a -A raw and ss -a -f link ? Whereas ss -a -A inet -f inet6 shows less sockets than ss -a -A inet? I think -f inet6 and -f inet are special cases, which are not documented properly.

(-0, -4 and -6 are aliases for -f link, -f inet, and -f inet6).

Did I mention that ss -A packet will show headings, but will never show any sockets? strace shows that it literally does not read anything. It seems to be because it treats packet sockets as always being "listening". ss does not bother to provide a warning about this. And this is different from raw sockets, which ss treats as being simultaneously "listening" and "non-listening".

(man 7 raw says that if a raw socket is not bound to a specific protocol which are not bound to a specific IP protocol, then it is transmit-only. I have not checked if these are treated as listening sockets only)