It is possible to call OOM-killer (out of memory killer) directly by keyboard combination:
SysRq-F
SysRq key is usually combined within PrtSc key on keyboards.
OOM-killer kills some process(-es) and system becomes responsive again.
Thx Raman for advice on this feature in comments above.
PS: This helped me a lot. I agree with opinion that this is the most useful advise about that problem if it caused by Chrome or whatever memory greedy software. But you need to keep in mind that OOM-killer could kill some really important process, use it carefully.
After some more investigation, it looks like this issue is less kernel related and more about how rsync and CIFS interact.
As far as I can make out, what is happening is that when rsync closes the destination file, CIFS (and probably any network filesystem) ensures the file is completely flushed and written to the remote disk before the close syscall returns. This is to assure any application that once the close operation completes successfully, the file has been completely saved and there is no risk of any further errors that could cause data loss.
If this wasn't done, then it would be possible for an application to close a file, exit thinking the save operation was successful, then later (perhaps due to a network problem) the data could not be written after all, but by then it is too late for the application to do anything about it, such as asking the user if they want to save the file somewhere else instead.
This requirement means that every time rsync finishes copying a file, the entire disk buffer must empty out over the network before rsync is allowed to continue reading the next file.
A workaround is to mount the CIFS share with the option cache=none which disables this feature, and causes all I/O to go direct to the server. This eliminates the problem and allows reads and writes to execute in parallel, however a drawback of this solution is that the performance is somewhat lower. In my case, network transfer speed drops from 110MB/sec to 80MB/sec.
This may mean that if you are copying large files, performance may well be better with the alternating read/write behaviour. With many smaller files, disabling the cache will result in fewer cache flushes each time a file is closed so performance may increase there.
It seems rsync needs an option to close its file handles in another thread, so it can start reading the next file while the last one is still being flushed.
EDIT: I have confirmed that cache=none definitely helps when transferring lots of small files (brings it from 10MB/sec up to 80MB/sec) but when transferring large files (1GB+) cache=none drops the transfer from 110MB/sec down to the same 80MB/sec. This suggests that the slow transfer from many small files is less about the source disk seeking, and more about having so many cache flushes from all the small files.
Best Answer
There are 5 tunables in the /proc file system to change linux' writeback behavior:
The configuration is quite complicated and documentation can be found at kernel.org. However, as jordanm already said, "Any userspace application can tell the kernel to write its dirty buffers to disk via the sync() system call." which means that any other process might render your configuration useless.
Also keep your Filesystem settings in mind: Mount options like noatime, data=writeback and nobarrier can dramatically improve your throughput but will also put your data at risk, if your disk controllers are not battery backed.