Recently there was a article about Firefox slowing down on videos due to the internal process that saves all open tabs every 10 seconds - you can increase this by going to about:config and increasing the timeout on browser.sessionstore.interval

Article

That however would not fix the problem with Opera.

Personally, I have a good graphics card and fast CPU and see random problems, slow downs and lag in various flash videos. I think the latest few releases are buggy and need refining as I never had this problem in a older version on a slower machine.

I hope the first solution helps you as I know it has helped a lot of people - just not me!

Lately, I've been experiencing video loading behavior issues again, even after applying what I suggested in my original answer.

Dash playback

From what I understand, the underlying issue is YouTube's implementation of Dynamic Adaptive Streaming over HTTP. From what I can tell, there are both server-side and client-side implementations at work here. The Wikipedia article linked above says:

Google's YouTube experimented with supporting MPEG-DASH on the server side. Google Chrome supports it on the client side. However, the implementation of the feature has resulted in video playback being severely degraded by various bugs, such as the video quality options being randomly greyed out and unselectable without multiple refreshes of the page.

Which sounds very familiar.

Solution

The currently very popular approach to solve this issue is to use the YouTubeCenter extension, which is available for a wide variety of browsers.

It allows you to disable Dash playback, but also notes:

Please note that by disabling Dash Playback will result in 480p and 1080p not being available. This is because of a recent YouTube change.

Disabling Dash playback

To disable Dash playback, just open the settings menu:

And untick the corresponding checkbox on the player page:

The result will be that videos will load as fast as possible and fully.

Please note, YouTubeCenter removes advertisements by default. If you want to support your favorite YouTube channels, be sure to consider disabling this feature.

Original answer:

Disabling the internal Flash plugin of Chrome

Seems like this is again a case of rubber duck debugging...

After posting this, I thought that I should probably check if I can simply disable the built-in Flash player in Chrome and see if that has an effect. The process is actually outlined in the Adobe Flash Player Help.

1. Grab the installer for the latest Flash Player from Adobe and install it.

2. Go to chrome://plugins/.

3. Click the Details button to expand the details about single plugins:

   

4. There should be 2 entries for the Adobe Flash Player. One is the built-in version in Chrome, the other is the normal installation. Disable the built-in one.

   

   You'll recognize the internal one easily, as it resides in the PepperFlash folder in the Chrome installation directory.

5. Now the caching behavior should be identical to the behavior observed in Firefox.

Please note that disabling the built-in version of Flash is usually discouraged.

Using the SmartVideo extension

Alex pointed out an extension called SmartVideo. This extension gives you more control over the buffering behavior of Chrome.

I tested it for a while and am quite pleased with the results. Especially the option Ensure that videos are buffered even if they are paused affects the undesired behavior as outlined in my question.

It has several advantages over disabling PepperFlash. First of all, it gives quite a few options which I didn't even have in mind before. Like preventing YouTube to auto-play videos.

Using PepperFlash is also an advantage when using multiple monitors. If you have a fullscreen video on one screen, and another Chrome window on the second, giving focus to the second Chrome window will still keep the fullscreen video in the foreground. In contrast, if you use the external Flash player, focusing the second Chrome window would pop the taskbar in front of the fullscreen video.