linuxshell-script
I'm trying to create a script which starts some programs
startup.sh
knetworkmanager
emesene
keepassx
The problem is that when I run the script, it only starts knetworkmanager. This is because it'll start it, then wait until it's finished. How can I start a program without waiting for it? I don't think I can just add "&" after each command, because all processes will still be killed when the script is finished.
The $! PID is that of su since you're backgrounding it rather than your $_CMD. You should move the & inside the closing quote.
Use pidof or pgrep to get your process' PID instead of using $!. If you're on a system that uses Upstart or has a similar daemon system, you should model your script after those and use the functions in /lib/lsb/init-functions or similar.
ulimit -v, and maybe set a hard or soft limit using the as option in limits.conf). This used to work well enough, but thanks to WebKit introducing gigacage, many gnome apps now expect unlimited address spaces and fail to run! sysctl vm.overcommit_memory, sysctl vm.overcommit_ratio, but this approach didn't work out for me.I also found:
wait and hope for some upstream patches to get into stable distro kernels. Also hope that distro vendors better tune kernel defaults and better leverage systemd cgroups to prioritise GUI responsiveness in desktop editions.
Some patches of interest:
So it's not just bad user space code and distro config/defaults that's at fault - the kernel could handle this better.
1) Disable the swap
Providing at least a small swap partition is recommended (Do we really need swap on modern systems?). Disabling swap not only prevents swapping out unused pages, but it might also affect the kernel's default heuristic overcommit strategy for allocating memory (What does heuristics in Overcommit_memory =0 mean?), as that heuristic does count on swap pages. Without swap, overcommit can still probably work in the heuristic (0) or always (1) modes, but the combination of no swap and the never (2) overcommit strategy is likely a terrible idea. So in most cases, no swap will likely hurt performance.
E.g., think about a long running process that initially touches memory for once-off work, but then fails to release that memory and keeps running the background. The kernel will have to use RAM for that until the process ends. Without any swap, the kernel can't page it out for something else that actually wants to actively use RAM. Also think about how many devs are lazy and don't explicitly free up memory after use.
3) set a maximum memory ulimit
It only applies per process, and a it's probably a reasonable assumption that a process shouldn’t request more memory that a system physically has! So it's probably useful to stop a lone crazy process from triggering thrashing while still being generously set.
4) keep important programs (X11, bash, kill, top, ...) in memory and never swap those
Nice idea, but then those programs will hog memory they're not actively using. It may be acceptable if the program only requests a modest amount of memory.
systemd 232 release has just added some options that make this possible: I think one could use the 'MemorySwapMax=0' to prevent a unit (service) like ssh having any of it's memory swapped out.
Nonetheless, being able to prioritise memory access would be better.
The linux kernel is more tuned for server workloads, so GUI responsiveness has sadly been a secondary concern... The kernel memory management settings on the Desktop edition of Ubuntu 16.04 LTS didn't seem to differ from the other server editions. It even matches the defaults in RHEL/CentOS 7.2 typically used as a server.
Swap thrashing (when the working set of memory, i.e. pages being read and writing to in a given short time-frame exceeds the physical RAM) will always lockup storage I/O - no kernel wizardry can save a system from this without killing a process or two...
I'm hoping Linux OOM tweaks coming along in more recent kernels recognise this working set exceeds physical memory situation and kills a process. When it doesn't, the thrashing problem happens. The problem is, with a big swap partition, it can look as if the system still has headroom while the kernel merrily over commits and still serves up memory requests, but the working set could spill over into swap, effectively trying to treat storage as if it's RAM.
On servers, it accepts the performance penalty of thrashing for a determined, slow, don't lose data, trade-off. On desktops, the trade off is different and users would prefer a bit of data loss (process sacrifice) to keep things responsive.
This was a a nice comical analogy about OOM: oom_pardon, aka don't kill my xlock
Incidentally, OOMScoreAdjust is another systemd option to help weight and avoid OOM killing processes considered more important.
I think "Make background writeback not suck" will help avoid some issues where a process hogging RAM causes another swap out (write to disk) and the bulk write to disk stalls anything else wanting IO. It's not the cause thrashing problem itself, but it does add to the overall degradation in responsiveness.
One problem with ulimits is that the accounting an limit applies to the virtual memory address space (which implies combining both physical and swap space). As per man limits.conf:
rss maximum resident set size (KB) (Ignored in Linux 2.4.30 and higher)
So setting a ulimit to apply just to physical RAM usage doesn't look usable anymore. Hence
as address space limit (KB)
seems to be the only respected tunable.
Unfortunalty, as detailed more by the example of WebKit/Gnome, some applications can't run if virtual address space allocation is limited.
Currently, it seems cumbersome, but possible to enable some kernel cgroup flags cgroup_enable=memory swapaccount=1 (e.g. in grub config) and then try use the cgroup memory controller to limit memory use.
cgroups have more advanced memory limit features then the 'ulimit' options. CGroup v2 notes hint at attempts to improve on how ulimits worked.
The combined memory+swap accounting and limiting is replaced by real control over swap space.
CGroup options can be set via systemd resource control options. E.g.:
Other useful options might be
These have some drawbacks:
In CGroup v2, they suggest that memory.high should be a good option to throttle and manage memory use by a process group. However this quote suggests that monitoring memory pressure situations needed more work (as of 2015).
A measure of memory pressure - how much the workload is being impacted due to lack of memory - is necessary to determine whether a workload needs more memory; unfortunately, memory pressure monitoring mechanism isn't implemented yet.
Given systemd and cgroup user space tools are complex, I haven't found a simple way to set something appropriate and leverage this further. The cgroup and systemd documentation for Ubuntu isn't great. Future work should be for distro's with desktop editions to leverage cgroups and systemd so that under high memory pressure, ssh and the X-Server/window manager components get higher priority access to CPU, physical RAM and storage IO, to avoid competing with the processes busy swapping. The kernel's CPU and I/O priority features have been around for a while. It seems to be priority access to physical RAM that's lacking.
However, not even CPU and IO priorities are appropriately set!? When I checked the systemd cgroup limits, cpu shares, etc applied, as far as I could tell, Ubuntu hadn't baked in any pre-defined prioritisations. E.g. I ran:
systemctl show dev-mapper-Ubuntu\x2dswap.swap
I compared that to the same output for ssh, samba, gdm and nginx. Important things like the GUI and remote admin console have to fight equally with all other processes when thrashing happens.
I wanted to enable hibernate, so I needed a big swap partition. Hence attempting to mitigate with ulimits, etc.
I put * hard as 16777216 in /etc/security/limits.d/mem.conf such that no single process would be allowed to request more memory than is physically possible. I won't prevent thrashing all together, but without, just a single process with greedy memory use, or a memory leak, can cause thrashing. E.g. I've seen gnome-contacts suck up 8GB+ of memory when doing mundane things like updating the global address list from an exchange server...
As seen with ulimit -S -v, many distros have this hard and soft limit set as 'unlimited' given, in theory, a process could end up requesting lots of memory but only actively using a subset, and run happily thinking it's been given say 24GB of RAM while the system only has 16GB. The above hard limit will cause processes that might have been able to run fine to abort when the kernel denies their greedy speculative memory requests.
However, it also catches insane things like gnome contacts and instead of loosing my desktop responsiveness, I get a "not enough free memory" error:
Unfortunately, some developers like to pretend virtual memory is an infinite resource and setting a ulimit on virtual memory can break some apps. E.g. WebKit (which some gnome apps depend on) added a gigacage security feature which tries to allocate insane amounts of virtual memory and FATAL: Could not allocate gigacage memory errors with a cheeky hint Make sure you have not set a virtual memory limit happen. The work-around, GIGACAGE_ENABLED=no forgoes the security benefits, but likewise, not being allowed to limit virtual memory allocation is also forgoes a security feature (e.g. resource control that can prevent denial of service). Ironically, between gigacage and gnome devs, they seem to forget that limiting memory allocation is itself a security control. And sadly, I noticed the gnome apps that rely on gigacage don't bother to explicitly request a higher limit, so even a soft limit breaks things in this case.
To be fair, if the kernel did a better job of being able to deny memory allocation based on resident memory use instead of virtual memory, then pretending virtual memory is unlimited would be less dangerous.
If you prefer applications to be denied memory access and want to stop overcommitting, use the commands below to test how your system behaves when under high memory pressure.
In my case, the default commit ratio was:
$ sysctl vm.overcommit_ratio
vm.overcommit_ratio = 50
But it only comes into full effect when changing the policy to disable overcommiting and apply the ratio
sudo sysctl -w vm.overcommit_memory=2
The ratio implied only 24GB of memory could be allocated overall (16GB RAM*0.5 + 16GB SWAP). So I'd probably never see OOM show up, and effectively be less likely to have processes constantly access memory in swap. But I'll also likely sacrifice overall system efficiency.
This will cause many applications to crash, given it's common for devs to not gracefully handle the OS declining a memory allocation request. It trades off the occasional risk of a drawn out lockup due to thrashing (loose all your work after hard reset) to a more frequent risk of various apps crashing. In my testing, it didn't help much because the desktop itself crashed when the system was under memory pressure and it couldn't allocate memory. However, at least consoles and SSH still worked.
How does VM overcommit memory work has more info.
I chose to revert to default for this, sudo sysctl -w vm.overcommit_memory=0, given the whole desktop graphical stack and the applications in it crash nonetheless.
Best Answer
When the shell that runs the script exits, it send a HUP signal to the processes you started. If those don't catch the signal, they will terminate. So whether just using & is sufficient depends on the application. To be safe, use nohup like that:
See the man pages for nohup and kill for more details or read the wikipedia article about nohup.