Sql-server – Replicate only specific records of a table

There is no supported way to achieve what you want to do, aside from complete reinitialization. That said, you might be able to achieve a shortcut and retain a supported configuration, but only under direction of a Microsoft support engineer. Attempting to modify the system replication tables without that guidance risks compounding your current problems.

My advice would be to open a case under your support contract, or take the hit of reinitializing and change operating practices to prevent a recurrence of the problem in future.

I personally DO NOT recommend setting up merge replication under such conditions. But, as there is no way for me (other than make this work or develop my own replication solution), I spent two weeks experimenting and found a solution.

1. Disable Precompute partitions option on Publication Properties -> Subscription Options -> Filtering window. This makes sp_MSmakegeneration lock only that particular table it's currently working on, not all merge replication affected tables. This option becomes enabled automatically if you have any row filtering. But actually, it's only intended to speed up parameterized filters (not my case). Note: turning this off will invalidate subscriptions; you must re-initialize them with fresh snapshot.
2. Create simple job with exec dbo.sp_MSmakegeneration command and schedule it every 1 minute. Normally, it called as part of synchronization process to enumerate local changes. But, if you skip calling it for 10-15 minutes (it means 20-30k changed/inserted rows in my case), the next synchronization calling it will block all client inserts/updates into replicated table; also, there is chances you will catch deadlock during its execution, making it impossible to run it successfully until server load is reduced. This, ofcourse, depends on your particular table indexes and usage.
3. Schedule your synchronization jobs to run frequently. I personally set it to run each 3 minutes. This reduces sync delay to 5-10 minutes. Strictly speaking, this is not mandatory, but syncing a lot of changes is quite a heavy load itself, so its easier for server to sync smaller parts.

Using this, I am able to successfully sync 3 servers, with 3-10k inserts/updates per minute on each during workday with no major delays.