OS Memory v. Ubuntu Memory v. Kernel Memory
You have asked to know how much the "Linux OS" is using (and not how much memory is being used by "applications"). "Linux" is just a kernel (albeit not a monolithic one; kernel modules contribute to the effective footprint). Many things that end users see as "the OS" are not part of the kernel, but part of the larger Ubuntu distribution. In a typical desktop installation, this includes the windowing system (commonly xorg), the window manager and desktop environment (Unity on Ubuntu by default, others include Gnome and KDE), as well as a whole host of other processes that perform essential operations so that you can interact with your system.
The word "applications", to end users, usually means things like "web browser", "spreadsheet", etc. I'll refer to these as "user applications".
There are ways to estimate your kernel size, but if you wish to come up with an estimate to how much memory Ubuntu is using, you would have to identify and add up all of the related processes (xorg, init, etc.). The other option is to use something like top when you have no other user applications running, and subtract the memory used by top and your terminal emulator and shell.
Total Memory
There are many ways, some more precise than others, to see a Linux system's memory usage. There are also many definitions of "memory in use". That said, for most non-technical uses, top(1) will do the job just fine:
top - 17:23:50 up 7 days, 19:15, 3 users, load average: 0.00, 0.01, 0.05
Tasks: 172 total, 2 running, 170 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.8 us, 0.1 sy, 0.0 ni, 99.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2049992 total, 1735420 used, 314572 free, 372420 buffers
KiB Swap: 2095100 total, 192 used, 2094908 free. 679116 cached Mem
As an added bonus, since the next question is often "what is using up the memory?", the next lines show the running processes and (among other things), how much virtual memory they are using.
Note that "free" memory on Linux is often a very small number, and that's a good thing. Free memory is essentially a wasted resource, so Linux instead allocates it to various caching uses. If an application needs that memory, it's still available; Linux is just putting it to good use in the meantime.
Best Answer
If you can modify the source code, then use rusage data to measure the RSS and record how many TCP connections are in play at the time of the measurement.
If source code cannot be changed, then use the RSS of the network app as reported by top or ps and get the number of network connections at the time of measurement from
lsof -i.Collect this data every minute while your application moves through peak load, and from that data you can come up with a formula that relates number of connections to RAM usage.
Of course there are a lot more things that you could measure, in particular you might want to measure kernel RAM usage, although tcp data structures should be predictable and calculable in advance. In any case, have a look at this question https://serverfault.com/questions/10852/what-limits-the-maximum-number-of-connections-on-a-linux-server for more information on TCP tuning and how to get a clear view of what is happening in the network stack.