Sql-server – SQL Server Memory Clerk Description

There are other memory consumers outside of the buffer pool. For instance, SQL Server will use memory for worker threads, multi-page allocations, linked servers, extended stored procedures, CLR, etc.

The buffer pool is never going to equate to 100% of memory consumed by SQL Server. It's a common misconception, as well as setting max memory will be the actual max memory SQL Server uses (not true).

EDIT: Here's how I think you could get non-buffer pool memory size. Note: wait for the input of other SQL Server professionals before taking this query as fact. I cannot guarantee that it is absolutely correct.

declare 
    @data_cache_size_mb decimal(12, 2),
    @plan_cache_size_mb decimal(12, 2),
    @total_mem_mb decimal(12, 2)

select
    @data_cache_size_mb = count(*) * 8 / 1024.
from sys.dm_os_buffer_descriptors

select
    @plan_cache_size_mb = sum(size_in_bytes) / 1024. / 1024.
from sys.dm_exec_cached_plans

select
    @total_mem_mb = sum(pages_allocated_count) * 8 / 1024.
from sys.dm_os_memory_objects

select
    @data_cache_size_mb as data_cache_size_mb,
    @plan_cache_size_mb as plan_cache_size_mb,
    @data_cache_size_mb + @plan_cache_size_mb as buffer_pool_size_mb,
    @total_mem_mb as total_instance_mem_mb,
    @total_mem_mb - (@data_cache_size_mb + @plan_cache_size_mb) as non_buffer_pool_size_mb

If a query does not have a memory grant then it cannot request memory from this 'workspace'. You can see which queries request memory in sys.dm_exec_query_memory_grants. As you already noticed only certain operators request memory grants, namely the operators that require some significant intermediate storage. A nested loop does not require any storage, as it operates one-row at a time, therefore it will not request a memory grant. A hash requires memory to built the hash tables, and a sort obviosuly requires sort tables. Queries without a grant still need some memory, but that adds up to trivial ammounts (maybe few KBs) and such memory is obtained form ordinary execution allocators. A grant is measured usually in MBs and can go into the 100s of MBs.

The memory reserved for grants is not exactlty 'part of the buffer pool', that makes it sound like is a separate portion of the buffer pool. Is more of a gray area. The memory grant is backed by the buffer pool. A query that is granted X amount of memory places a reservation on the BP. It does not 'steal' the pages from BP right away. But since the (unstolen part of) BP contains data pages, the BP can make good on the reservation promise because it can evict data pages to disk anytime it likes. As the query makes progress it may request pages from the BP from the grant it acquired (from the reservation) and as the BP fulfills the request, the pages handed out to the query become really 'stolen', ie. the BP can no longer use them for data until the query release them back. In order to ensure all grants can be honored the sum of reservations cannot exceed the BP size (ie. there is no VM style over-allocation). These grant reservations tend to err on the overallocation side by a generous margin as things can turn out quite ugly if a query consumes all the grant and styll needs memory: spills occur (extremly slow) or the query may give up and error.

See Understanding SQL server memory grant