Ubuntu – Wifi broadcom 4313 with proprietary driver (Ubuntu) randomly stops connecting to wireless networks

I have a laptop with Broadcom 4313 wireless module, using Ubuntu 13.10 and Windows 7 as OS

When reconnecting to network or when waking laptop from suspend/rebooting it sometimes cannot connect to any wireless network. Disable/re-enable network doesn't help. I have to reboot laptop few times in a row to make it work again.

It throws these errors to syslog:

wpa_supplicant[1091]: wlan0: Authentication with xx:xx:xx:xx:b4:80 timed out.
NetworkManager[814]: <info> (wlan0): supplicant interface state: associating -> disconnected
NetworkManager[814]: <info> (wlan0): supplicant interface state: disconnected -> scanning
wpa_supplicant[1091]: wlan0: Trying to associate with xx:xx:xx:xx:b4:80 (SSID='xxxxxxxxx' freq=2457 MHz)
NetworkManager[814]: <info> (wlan0): supplicant interface state: scanning -> associating
wpa_supplicant[1091]: wlan0: CTRL-EVENT-ASSOC-REJECT status_code=16
wpa_supplicant[1091]: wlan0: Authentication with xx:xx:xx:xx:b4:80 timed out.

The only solution to have any wireless connection is to switch from proprietary driver wl to opensource brcmsmac (or reboot few times), the the second one has so poor connection even if sitting next to wireless router that it becomes unusable in most situations.

Btw, sometimes it even happens on Windows, but it has some 'troubleshooting' routine which somehow makes it work then.

Does anyone know why this is happening?

Is there any way to properly flush networking so it will think that system has just rebooted? service networking restart doesn't fit because it breaks X server somehow.

Google doesn't give me any clue how to fix it.