Why do Application Shortcuts in System Preferences disappear

I think that's a bug with Finder. Services that receive folders as input don't seem to be listed in the Services menu when folders are selected (but it only applies to column view; they are listed in other view modes).

In any case, one alternative would be to just use an AppleScript without wrapping it as a service. You can give it an app-specific shortcut with FastScripts.

try
    tell application "Finder"
        open (get selection) using path to application "TextMate"
    end tell
end try

1. Is there a way to see what key combination is already taken, and for what purpose?

You can see if there is a conflict with your chosen shortcut. From "If a keyboard shortcut doesn’t work":

1. Choose Apple menu > System Preferences, and then click Keyboard.
2. Click Keyboard Shortcuts. A yellow warning triangle appears next to conflicting keyboard shortcuts.

2. Is there a key combination convention or rule for the different modifier keys? What key would be appropriate for what task? (e.g. when would I choose Command or Shift or Option or Ctrl or one of the function keys)

For dictionary lookups, Apple has already defined Ctrl+Cmd+D as "Display the definition of the selected word in the Dictionary app."

The OS X Human Interface Guidelines on Keyboard Shortcuts states the following about when to use these modifier keys and for what purposes. The recommended priority is to use Cmd frequently, use Option sparingly, avoid using Ctrl and use Shift for complementary functions of a common command (of course, you may observe that in certain instances Apple does not follow these guidelines).

Quote from the Keyboard Shortcuts page:

As much as possible, use the Command key as the main modifier key in a keyboard shortcut. For example, Command-P uses Command to modify the P key. For a command that complements another more common command, you can add Shift to the shortcut. For example, the shortcut for the complementary Page Setup command adds Shift to the shortcut for Print to give Shift-Command-P.

Use the Option key sparingly. If there’s a third, less common command that’s related to a pair of commands that use Command and Shift-Command, you can use Option-Command for the third command’s keyboard equivalent. Use combinations like these very rarely. You can also use Option for a keyboard shortcut that’s a convenience or power-user feature. For example, the Finder uses Option-Command-W for Close All Windows and Option-Command-M for Minimize All Windows.

As much as possible, avoid using the Control key. Because the Control key is already used by some of the universal access features—as well as in Cocoa text fields where Emacs-style key bindings are often used—it should be used as a modifier key only when necessary.

List multiple modifier keys in the correct order. If you use more than one modifier key in a shortcut, always list them in this order: Control, Option, Shift, Command.