Sql-server – USERSTORE_TOKENPERM growing very fast

I'm guessing the 'memory left' output you posted is from Christian Bolton's VAS usage analysis script. Is that correct? If so, you have only 17-18 MB available in the memtoleave area, which is likely to cause a problem at some point and could cause all or most of the error messages that you pasted.

This query from Jonathan Kehayias here will give you an idea of what is using memory in the memtoleave area:

SELECT type, virtual_memory_committed_kb, multi_pages_kb 
FROM sys.dm_os_memory_clerks 
WHERE virtual_memory_committed_kb > 0 OR multi_pages_kb > 0

The post also explains memtoleave well.

Whether the size of the caches you asked about is a problem depends on a number of things. Since this is a 32-bit instance, one of the more relevant things to consider is whether the allocations for those caches are allocated using the single-page allocator, which uses the buffer pool, or the multi-page allocator, which prior to SQL Server 2012 doesn't use the buffer pool and instead consumes memory in the memtoleave area. You have about 13 GB allocated to the buffer pool, so a few hundred MB allocated to this or that isn't necessarily going to cause a problem. However, a few hundred MB in the memtoleave area is enough to cause some of the errors you listed.

Kehayias' post explains these concepts as well as your options for solving the problem if it is indeed memtoleave exhaustion.

I would focus mainly on what is starving the memtoleave area, as well as what overall memory availability in Windows looks like, along with general measures of memory availability in SQL Server such as page life expectancy.

I would also consider whether the problem helps make a business case for migrating to a 64-bit version of SQL Server, as this avoids memtoleave issues unless the whole server is simply running out of memory. Understanding and managing SQL Server's memory usage is significantly simpler on 64-bit builds, especially on SQL Server 2012 where all allocations go through the buffer pool. You may also want to consider applying SP4 as there are a number of known bugs in SP3.

To start with I must say you have set max server memory to 6 GB and total memory is 8 GB so you have just left 2 GB for the OS, which in many cases, even if nothing is installed apart from SQL Server on a Windows machine, is too little memory provided to OS. To function properly, on a system with antivirus installed, OS must be given 4 GB at least. I leave 2GB for OS straight away and 1.5 G for AV.

The Target Server Memory is sometimes 6GB and then drops back to Total Server Memory (approx 5.3GB, never reaches 6GB).

Target server memory signifies how much memory is required by SQL Server to function properly in the ideal case. Target server memory is trying to be 6 GB because you have set the max server memory value to 6 GB. It's trying to consume all the memory it is allowed to.

Total server memory is what SQL Server is actually able to consume right now. This is committed memory and backed by physical RAM. This is 5.5 GB max in your case.

SQL Server is trying to grow its memory consumption but after reaching 5.3 or 5.5 GB, the OS is asking SQL Server to not grow its memory consumption further and might be actually flagging low memory notification. This is happening because OS might be facing low memory as already said above. SQLOS responds if Windows OS faces memory pressure by asking its caches its trim down their consumption. You can Query the Ring Buffer to check if there was low memory notification signaled. I must add DMV sys.dm_os_ring_buffer is undocumented but safe.

I see that pages are dropped from the cache - but there is still 700MB of memory left. If nothing needed the memory, how would you explain the fact that pages are removed from cache? I would expect that SQL Server only removes pages when it needs memory.

If you are looking for free memory, I would not suggest you to look at the DMV sys.dm_os_buffer_descriptors. The OS counter Available Mbytes will tell you the amount of physical memory, in bytes, available to processes running on the computer. I suggest you to also see What is a deterministic method for evaluating a sensible buffer pool size? and also read Does SQL Server need more RAM to find out how much RAM SQL Server needs and if SQL Server is facing memory pressure. From what you mentioned, if you are sure pages are getting removed from buffer pool then yes SQL Server feels that pages have to be moved because it needs space to accommodate new pages. I am not sure how you calculated the 700 MB free.

One other thing, please don't look at Task Manager for SQL Server memory consumption. It does not always give you the correct value especially when the SQL Server service account has the lock pages in memory privilege. In your case, even if SQL Server has max server memory of 6 GB, the OS being given just 2 GB, which is forcing SQL Server to not grow its consumption because 2 GB is low for SQL Server. Is there anything apart from SQL Server running on the system?

If you want to calculate SQL Server memory consumption please use:

select
(physical_memory_in_use_kb/1024) Physical_Memory_usedby_Sqlserver_MB,
(locked_page_allocations_kb/1024 ) Locked_pages_used_Sqlserver_MB,
(virtual_address_space_committed_kb/1024 ) Total_Memory_in_MB,--RAM+ Pagefile
process_physical_memory_low,
process_virtual_memory_low
from sys.dm_os_process_memory

What I don't understand is why Total_Memory_in_MB isn't equal to 6144 (max memory).

The column Total_Memory_in_MB signifies total memory used by SQL Server (RAM+page file). The RAM is actually physical memory used or committed memory. Some part of SQL Server process is also paged to disk and that constitutes as as virtual memory or page file and so if you are going to see TOTAL memory consumed by SQL Server it would be sum of physical memory and the Page file.

While the column Physical_Memory_usedby_Sqlserver_MB is just the physical memory (memory backed by physical RAM or committed memory) used. This is the reason why both are different. If you see the real column first one is Physical memory used and other one is Virtual memory committed.

If you want to see paged memory that would be the difference between Total_Memory_in_MB and Physical_Memory_usedby_Sqlserver_MB.

NOTE: Total memory used would be greater than physical memory used.