Quick Look won't let you do this. Mainly because the core audio/video stuff behind quicktime doesn't support it.
The easiest way to do this would honestly be to use VLC. This supports both file types you mention.
The other way is to check out Perian, which gives you a bunch of extra codecs.
Personally, I would try the VLC way first. You won't be able to view these items with quickview or preview.app, but VLC handles pretty much any file type I have ever thrown at it.
Best Answer
One way that I know of, without dragging the window back, is to in Terminal use the
killall Finder
command.This works for me in macOS Catalina.