This isn't a complete answer, as I admit I don't know the complete answer, but can't fit into a comment.
Short detour - regarding SQLOS memory there is a wealth of information coming from a guy that worked on the internals of SQL Server storage team - Slava Oks. He has lots of articles on MSDN blogs regarding the SQLOS memory manager and they are all worth a read. Now I know we're on to SQL Server 2012, but the info is valuable nonetheless.
Now, regarding your first question, if there's a limit of max memory taken by db page cache before being cleared out and how is it controlled, I'll quote his word on it.
From article SQLOS's memory manager and SQL Server's Buffer Pool:
Buffer Pool commits pages on demand. Depending on internal memory requirements and external memory state, it calculates its target, amount of memory it thinks it should commit before it can get into memory pressure. To keep system out of paging target is constantly recalculated. Target memory can't exceed max memory that represents max server memory settings. Even if you set min server memory equal to max server memory Buffer Pool will only commit its memory on demand. You can observe this behavior by monitoring corresponding profiler event.
From this article's comment I understand that SQL Server uses a LRU algorithm (LRU = least recently used). If you really need to go in depth, SQL Server's LRU algorithm in particular is described in the following research paper: The LRU-K Page Replacement Algorithm For Database Disk Buffering.
Now, for your second question, the log caches are not really exposed so they can't be changed/tuned, as they are a part of the SQLOS internal memory structure. Quoting from an answer from MSDN forum:
Essentially each database has a log buffer/cache for its transaction
log which is a small contiguous memory allocation, that is 60K in size
and used by active transactions, and flushed when a transaction
commits, or when the buffer space fills up. Multiple active
transactions can be interleaved in the log buffer, and if any one
transaction commits, the buffer memory is written to the transaction
log file by the log writer. If the space fills up, the contents get
flushed to disk allowing the buffer/cache memory to be reused. This
is why log IO can range for 512bytes to 64K (its actually 60K), but it
is always sequential so varying write sizes doesn't matter that much.
The format of the log records is different than data pages as
documented in the BOL
(http://msdn.microsoft.com/en-us/library/ms190925.aspx), so the size
of a log record is variable, but the size of the log buffer/cache is
consistent.
We had to bring MS in on this one via a paid support call. After a week of dumps, xperf analysis, driver updates, playig with Virtual infrastructure we ruled out all the basics (or so we thought) - over a week later we found the problem - there was over 500,000 sys.server_event_notifications defined - these all looked as follows:
name object_id parent_class parent_class_desc parent_id create_date modify_date service_name broker_instance creator_sid principal_id
SQLWEP_E0B9EA66_1E73_45B7_B528_75166DC7A15D 337640 100 SERVER 0 2013-06-14 01:56:27.077 2013-06-14 01:56:27.077 SQL/Notifications/ProcessWMIEventProviderNotification/v1.0 706DF30B-6BD5-458B-85A4-F1C174263ADE 0x010100000000000512000000 259
SQLWEP_6492D14E_99D2_49F2_9CD9_43F4B7DC7F63 338711 100 SERVER 0 2013-02-12 17:50:55.897 2013-02-12 17:50:55.897 SQL/Notifications/ProcessWMIEventProviderNotification/v1.0 706DF30B-6BD5-458B-85A4-F1C174263ADE 0x010100000000000512000000 259
SQLWEP_C9871B51_F866_48A6_862E_08EC4F92DAFA 351457 100 SERVER 0 2013-03-06 02:04:18.070 2013-03-06 02:04:18.070 SQL/Notifications/ProcessWMIEventProviderNotification/v1.0 706DF30B-6BD5-458B-85A4-F1C174263ADE 0x010100000000000512000000 259
SQLWEP_0E5D8FAF_1A9C_43FA_8EDF_A5713A8C7825 362061 100 SERVER 0 2014-02-08 04:57:17.217 2014-02-08 04:57:17.217 SQL/Notifications/ProcessWMIEventProviderNotification/v1.0 706DF30B-6BD5-458B-85A4-F1C174263ADE 0x010100000000000512000000 259
SQLWEP_712ED252_E529_4022_9EA3_C785BAEB8147 364203 100 SERVER 0 2013-04-05 15:52:11.190 2013-04-05 15:52:11.190 SQL/Notifications/ProcessWMIEventProviderNotification/v1.0 706DF30B-6BD5-458B-85A4-F1C174263ADE 0x010100000000000512000000 259
SQLWEP_E2D31BC2_E416_4D12_8B4A_88F4183FE021 374807 100 SERVER 0 2014-03-15 13:31:25.390 2014-03-15 13:31:25.390 SQL/Notifications/ProcessWMIEventProviderNotification/v1.0 706DF30B-6BD5-458B-85A4-F1C174263ADE 0x010100000000000512000000 259
SQLWEP_7B389ACB_EA1F_434C_8B2A_7B93D4D0F098 376949 100 SERVER 0 2013-05-12 05:02:12.450 2013-05-12 05:02:12.450 SQL/Notifications/ProcessWMIEventProviderNotification/v1.0 706DF30B-6BD5-458B-85A4-F1C174263ADE 0x010100000000000512000000 259
SQLWEP_4A828CDE_D72D_4277_97C0_030686F86377 378020 100 SERVER 0 2013-01-21 16:49:04.780 2013-01-21 16:49:04.780 SQL/Notifications/ProcessWMIEventProviderNotification/v1.0 706DF30B-6BD5-458B-85A4-F1C174263ADE 0x010100000000000512000000 259
SQLWEP_4EA3C2AD_ACCE_4A66_83E9_0676D2250C34 387553 100 SERVER 0 2014-04-24 20:26:31.053 2014-04-24 20:26:31.053 SQL/Notifications/ProcessWMIEventProviderNotification/v1.0 706DF30B-6BD5-458B-85A4-F1C174263ADE 0x010100000000000512000000 259
SQLWEP_6EA3C0FB_6EDA_4B9D_A3FE_D2B07799212C 389695 100 SERVER 0 2013-06-16 18:07:06.043 2013-06-16 18:07:06.043 SQL/Notifications/ProcessWMIEventProviderNotification/v1.0 706DF30B-6BD5-458B-85A4-F1C174263ADE 0x010100000000000512000000 259
These had all been created gradually over 5 years, what they are, where they come from, no one could say & MS couldn't find any related alerts to justify them either. Dropping 1 notification took 4 seconds so we would have been looking at 22 days to drop them all - suffice to say we ended up nuking the SQL instance & re-installing SQL from scratch.
Best Answer
It was tempdb contention. Adding more tempdb data files and enabling trace flag 1118 solved the issue.