Linux – “screen is terminating” for non-root

Although this is a relatively old question, the answer is still the same. You have a virtual machine (running on a physical host) and some sort of storage (either shared storage – a FC SAN, iSCSI storage, an NFS share – or local storage).

With virtualisation, many virtual machines try to access the same physical resources at the same time. Due to physical limitations (number of read/write operations – IOPS; throughput; latency) there might be a problem to satisfy all storage requests of all physical machines at the same time. What usually happens: you will be able to see "SCSI retries" and failed SCSI operations in the operating systems of your virtual machines. If you get too many errors/retries in a certain period of time, the kernel will set the mounted filesystems read-only in order to prevent damage to the filesystem.

To cut the long story short: Your physical storage is not "powerful" enough. There are too many processes (virtual machines) accessing the storage system at the same time, your virtual machines do not get the response from the storage fast enough, and the filesystem goes read-only.

There are not terribly many things you can do. The obvious solution is better/additional storage. You can also modify the parameters for SCSI timeouts in the Linux kernel. Details are described, e.g., in:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1009465

http://www.cyberciti.biz/tips/vmware-esx-server-scsi-timeout-for-linux-guest.html

However, this will only "postpone" your problems, because the kernel only gets more time before the filesystem will be set read-only. (I.e., you do not solve the cause of the problem.)

My experience (several years with VMware) is that this problem only exists with Linux kernels (we're using RHEL and SLES) and not with Windows servers. Also, this problem occurs on all sorts of storage – FC, iSCSI, local storage. For us, the most critical (and expensive) component in our virtual infrastructure is storage. (We're now using HP LeftHand with 1 Gbps iSCSI connections, and have not had any storage issues ever since. We chose LeftHand (over traditional FC-solutions) for its scalability.

A process in S or D state is usually in a blocking system call, such as reading or writing to a file or the network, waiting for a called program to finish, or while waiting on semaphores or other synchronization primitives. It will go into the sleep state while waiting.

You can't "wake it up" - it will only proceed when the data/resource it is waiting for becomes available. This is all normal and expected, and only a problem when trying to kill it.

You can try and use strace -p pid to find out which system call is currently happening for process pid.

From wikipedia :

An uninterruptible sleep state is a sleep state that won't handle a signal right away. It will wake only as a result of a waited-upon resource becoming available or after a time-out occurs during that wait (if specified when put to sleep). It is mostly used by device drivers waiting for disk or network IO (input/output). When the process is sleeping uninterruptibly, signals accumulated during the sleep will be noticed when the process returns from the system call or trap.

A process blocked in a system call is in uninterruptible sleep, which as its name says, is really uninterruptible even by root.

Normally, processes cannot block SIGKILL. But kernel code can, and processes execute kernel code when they call system calls, during which Kernel code blocks all signals. So if a system call blocks indefinitely, there may effectively be no way to kill the process. The SIGKILL will only take effect whenever the process completes the system call.