I have an iMac with this spec:
I have two external monitors, both 4k. Recently, Spaces has become almost unusably slow for me. Switching between spaces on the main display (usually between full-screen IntelliJ projects) using the trackpad is the laggiest. It will scroll a bit, pause for a second, then jerkily scroll the rest of the way. I think I managed to speed it up a little by removing files from my desktop, but that certainly hasn't fixed the problem.
If I open Spaces, then dragging them to rearrange them takes several attempts, because the drag usually fails to start (or at least takes longer than the 30 seconds or so I've tried waiting), and even hovering over different windows takes a couple of seconds before moving the blue highlight.
For some reason, things seem smoother on the external monitors, which only have about two applications each on them.
Most things, like dragging windows around and opening menus, happen just as smoothly as you'd expect them to.
Best Answer
I may not be able to pin down exactly what the bottleneck is, but in effect - OS X tracks every single window opened across all apps and then has to lay them out for each space and decide if a window is visible, occluded by another window or in the dock or hidden entirely by Finder hiding an app.
All of this calculation is pushed to the GPU for buttery smooth animation. At some point, you will exhaust the resources on the GPU whether it's the amount of pixels being stored, the total size of each window, the number of spaces and calculations to be made, transfer speeds from the CPU / RAM to the caches on the GPU. When you hit that bottleneck, either the GPU can't be as efficient or it rejects the work entirely and then you see the CPU doing the rendering.
My suspicion is you are getting the CPU to render which would explain the dramatic drop off in responsiveness. Also - you are pushing somewhere in the order of (5120x2880) + ( 2 x (4096 x 2034) ) = 31,408,128 pixels
If you can make this happen on command - I bet an Apple Engineer would love to look over
sudo sysdiagnose WindowServerwhen the machine is performing miserably. A call to AppleCare or a visit to a genius bar would be the step to get engineering involved.In the mean time - you might be able to experiment with reducing the load on the GPU by:
For your 4k external displays - are you in MST or SST mode? - https://support.apple.com/en-us/HT206587 - if you went to SST, that might alleviate the bottleneck or require less concessions on the window count at the expense of a 30 Hz refresh rate. The 5k iMac is listed as only supporting one 4k display in MST mode, so that could be an explanation the issue you are seeing.