How to clear the application used to open a file, i.e. set it to none

There is not a safe and advisable way to achieve what you are after because this function of the OS is controlled by the Launch Services database. Developers register the filetypes their applications support with the OS on first launch.

The best way to get this fixed is to petition the developer to add support for the file extension in a future version of their application.

With that said you have already found the only documented way that Apple offers for overriding the edicts of the Launch Services database. If you want to dig deeper and take the road less traveled supported here are a few pointers.

There is a .plist file that governs part of what Launch Services does located at: ~/Library/Preferences/com.apple.LaunchServices.plist

Apple notes that removing this plist can clear out unwanted Open With contextual menu options. But that's the opposite of what you're after.

It's a binary .plist but you could convert it to an xml version and experiment a little to see if you can come up with a recipe to achieve what you're after. This isn't advisable, it could cause harm, and it will most likely get overwritten by the OS at some point.

There is also a tool hidden a bit deeper in the OS called lsregister which could be helpful in exploring Launch Services's plumbing. Here is a helpful explanation of lsregister and its options.

From Apple’s Launch Services Programming Guide (all emphasis mine):

All applications available on the user’s system must be registered to make them known to Launch Services and copy their document binding and other information into its database. It isn’t ordinarily necessary to perform this task explicitly, since a variety of utilities and services built into the Mac OS X system software take care of it automatically:

• A built-in background tool, run whenever the system is booted or a new user logs in, automatically searches the Applications folders in the system, network, local, and user domains and registers any new applications it finds there. (This operation is analogous to “rebuilding the desktop” in earlier versions of Mac OS.)
• The Finder automatically registers all applications as it becomes aware of them, such as when they are dragged onto the user’s disk or when the user navigates to a folder containing them.
• When the user attempts to open a document for which no preferred application can be found in the Launch Services database, the Finder presents a dialog asking the user to select an application with which to open the document. It then registers that application before launching it.

In spite of these automatic registration utilities, it may sometimes be necessary to register an application explicitly with Launch Services. For example, although developers are encouraged to package their applications so that they can be installed by simply dragging them onto the user’s disk, some applications may require more elaborate custom installer software. In such cases, the installer should call one of the Launch Services registration functions LSRegisterFSRef or LSRegisterURL to register the application explicitly.

Note the API calls needed by the only named manual registration procedure (source not available on opensource.apple.com, I’m afraid).

While working around a bug in processing of Launch Services on Leopard with FileVault enabled, I noticed that ~/Library/Preferences/com.apple.LaunchServices.plist is:

• processed only on login after boot, as input data to the buildup of the Launch Services database proper (FileVault-enabled Leopard often failed to do this step, resulting in seemingly lost user settings); and

• cached as long as the machine is not rebooted.

Simply put, it’s the user domain persistence layer of Launch Services, and modifications to that persistence layer are only acknowledged on next processing – reboot or reseed.