Linux – In linux does “/proc/sys/net/ipv4/tcp_keepalive_time” has impact on both client & server

It's part of the TCP (or UDP, etc.) header, in the packet. So the server finds out because the client tells it. This is similar to how it finds out the client's IP address (which is part of the IP header).

E.g., every TCP packet includes an IP header (with source IP, destination IP, and protocol [TCP], at least). Then there is a TCP header (with source and destination port, plus more).

When the kernel receives a SYN packet (the start of a TCP connection) with a remote IP of 10.11.12.13 (in the IP header) and a remote port of 12345 (in the TCP header), it then knows the remote IP and port. It sends back a SYN|ACK. If it gets an ACK back, the listen call returns a new socket, set up for that connection.

A TCP socket is uniquely identified by the four values (remote IP, local IP, remote port, local port). You can have multiple connections/sockets, as long as at least one of those differs.

Typically, the local port and local IP will be the same for all connections to a server process (e.g. all connections to sshd will be on local-ip:22). If one remote machine makes multiple connections, each one will use a different remote port. So everything but the remote port will be the same, but that's fine—only one of the four has to differ.

You can use, e.g., wirehsark to see the packet, and it'll label all the data for you. Here is the source port highlighted (notice it highlighted in the decoded packet, as well as the hex dump at the bottom):

I've found the answer while still writing the question. I've decided to post it anyway because others may find this insightful, and then answer it myself; I hope this is not frowned upon :)

The user Philipp Matthias Hahn on the linux-kernel mailing list has figured it out at least partially:

As far as I researched for IPv4 some time ago, the "default" value gets
copied to newly created interfaces only once.
"all" on the other hand allways gets applied in addition to the current
setting, but it depends on the exact setting, if its ORed, ANDed, or
whatevered:
    log_martians         OR
    accept_redirects     AND
    forwarding           ?
    mc_forwarding        AND
    medium_id
    proxy_arp            OR
    shared_media         OR
    secure_redirects     OR
    send_redirects       OR
    bootp_relay          AND
    accept_source_route  AND
    rp_filter            AND
    arp_filter           OR
    arp_announce         MAX
    arp_ignore           MAX
    arp_accept
    app_solicit
    disable_policy
    disable_xfrm
    tag
(see include/linux/inetdevice.h:83 for IN_DEV_{AND,OR,MAX}CONF)

Putting a new value in "all" doesn't change the value you read from
"$interface", but it only gets computed and used internally.

He doesn't cover accept_ra but at least it's clear now how all and default work, or rather, how they do not work as I would have expected.