File descriptors and /dev/fd

The file descriptor 1 translates to the stdout FILE structure in the Kernel's Open Files Table.

This is a misunderstanding. The kernel's file table has nothing whatsoever to do with user-space file structures.

In any event, the kernel has two levels of indirection. There is the internal structure that represents the file itself, which is reference counted. There is an "open file description" that is reference counted. And then there is the file handle, which is not reference counted. The file structure points the way to the inode itself. The open file description contains things like the open mode and file pointer.

When you call close, you always close the file handle. When a file handle is closed, the reference count on its open file description is decremented. If it goes to zero, the open file description is also released and the reference count on the file itself is decremented. Only if that goes to zero is the kernel's file structure freed.

There is no chance for one process to release a resource another process is using because shared resources are reference counted.

At first I tried tracing a few xterms back to the xterm pid based on info I found in /proc/locks but it was loose. I mean, it worked, I think, but it was at best circumstancial - I don't fully understand all of the information that file provides and was only matching what seemed to correspond between its content and known terminal processes.

Then I tried watching lsof/strace on an active write/talk process between ptys. I had never actually used either program before, but they seem to rely on utmp. If my targeted pty did not have a utmp entry for whatever reason they both refused to admit that it existed. Maybe there's a way around that, but i was confused enough to abandon it.

I tried some udevadm discovery with 136 and 128 major number device nodes as advertised for pts and ptm respectively in /proc/tty/drivers, but I also lack any very useful experience with that tool and once again turned up nothing substantial. Interestingly, though, I noticed the :min range for both device types was listed at a staggering 0-1048575.

It wasn't until I revisited this this kernel doc that I started thinking about the problem in terms of mounts, though. I had read that several times before but when continued research in that line led me to this this 2012 /dev/pts patchset I had an idea:

sudo fuser -v /dev/ptmx

I thought what do I usually use to associate processes with a mount? And sure enough:

                     USER        PID ACCESS COMMAND
/dev/ptmx:           root      410   F.... kmscon
                     mikeserv  710   F.... terminology

So with that information I can do, for instance from terminology:

sudo sh -c '${cmd:=grep rchar /proc/410/io} && printf 1 >/dev/pts/0 && $cmd'
###OUTPUT###
rchar: 667991010
rchar: 667991011

As you can see, with a little explicit testing such a process could be made to pretty reliably output the master process of an arbitrary pty. Regarding the sockets, I'm fairly certain one could approach it from that direction as well using socat as opposed to a debugger, but I've yet to straighten out how. Still, I suspect ss might help if you're more familiar with it than I:

sudo sh -c 'ss -oep | grep "$(printf "pid=%s\n" $(fuser /dev/ptmx))"'

So I set it up with a little more explicit testing, actually:

sudo sh <<\CMD
    chkio() {
        read io io <$1
        dd bs=1 count=$$ </dev/zero >$2 2>/dev/null
        return $((($(read io io <$1; echo $io)-io)!=$$))
    }
    for pts in /dev/pts/[0-9]* ; do
        for ptm in $(fuser /dev/ptmx 2>/dev/null)
            do chkio /proc/$ptm/io $pts && break
        done && set -- "$@" "$ptm owns $pts"
    done
    printf %s\\n "$@"
 CMD

It prints $$ num \0 null bytes to each pty and checks each master process's io against a previous check. If the difference is $$ then it associates the pid with the pty. This mostly works. I mean, for me, it returns:

410 owns /dev/pts/0
410 owns /dev/pts/1
710 owns /dev/pts/2

Which is correct , but, obviously, it's a little racy. I mean, if one of those others was reading in a bunch of data at the time it would probably miss. I'm trying to figure out how to change the stty modes on another pty in order to send the stop bit first or something like that so I can fix that.